Deep Neural Network Method Research Articles

Use of ResNets for HLB Disease Detection on Orange Leaves Using Terrestrial Multispectral Images

Abstract. Huanglongbing (HLB) is a bacterial disease transmitted by different vectors of sap-sucking insects. It affects all crops of citrus trees, decreasing the values of those fruits in the market and eventually the decay of orchards. In Brazil, the world's leading orange producer, citriculture faces severe issues with HLB and substantial economic loss. Technical means of scanning the orchards with high-throughput becomes essential for the sustainability of this industry. In this study, we propose to investigate an operational strategy consisting of scanning large portions of foliage (the canopy of one tree or more) in which there can few early foliage symptoms. It is proposed to investigate deep learning tools to solve this complex binary classification problem. The study is based on a dataset comprising 1,297 terrestrial multispectral (14 channels) images captured at high spatial resolution in a commercial orange orchard in Brazil. It is proposed to adapt and retrain standard neural network architectures, namely ResNets18 and ResNets34, to process such images. Our analysis reveals promising results, with both models demonstrating convergence and achieving stable performance. Notably, ResNet18 outperformed ResNet34, achieving an accuracy of 76.45% compared to 66.79% from ResNet34. These findings suggest that deep neural network methods can effectively manage non-radiometrically calibrated data and accurately distinguish images with HLB symptoms from healthy plants . However, with reduced datasets and limited possibilities for transfer learning and fine-tuning, it seems that only reasonable sized networks can be trained. Thus, more advanced state-of-the-art tools of the are still challenging to deploy for agricultural multi- or hyperspectral data.

Combining long short-term memory and shape superposition for estimating moving load-induced global responses of bridges

Recently, a method of estimating the global responses of bridges was introduced, by combining a deep neural network (DNN) and shape superposition method (SSM). This DNN-SSM model demonstrated superior performance to the traditional estimation method based on the least-squares method. However, the application and validation of this approach have been restricted to static problems. For bridges, the dynamic effect induced by the moving load requires consideration. Therefore, this study proposes a long short-term memory (LSTM)-SSM model to improve the estimation performance for dynamic global responses. The LSTM-SSM model consists of LSTM layers, followed by fully connected DNN layers and a post-process part based on SSM. Limited displacement, slope, and strain data were used as the inputs, and the global displacement, slope, and strain were estimated. To validate the effectiveness and improvement of the LSTM-SSM model, a numerical model of a cable-stayed bridge was used to compare the proposed model with the previously developed DNN-SSM model. The validation results indicated that the LSTM-SSM model can provide more accurate estimates of the dynamic global response than DNN-SSM models. Finally, a technique for structural safety monitoring based on the proposed method was introduced.

Solving the Inverse Potential Problem in the Parabolic Equation by the Deep Neural Networks Method

Solving the Inverse Potential Problem in the Parabolic Equation by the Deep Neural Networks Method

Research on Explainability of Deep Neural Networks and Its Applications

In recent years, Artificial Intelligence (AI) models have surged in popularity and success, garnering much attention across many disciplines and applications. In particular, recent developments in Deep Neural Networks (DNN) have allowed them to match and even surpass human capabilities in many tasks. However, the complexity of deep neural networks causes them function like black boxes that are very difficult for humans to understand, thus making them untrustworthy and limiting their usefulness in high-risk applications. Explainable Artificial Intelligence aims to alleviate this issue by generating explanations for DNNs to allow humans to understand their reasoning and increase their transparency. This paper reviews research on neural network explainability, detailing its definition, necessity, classification, and evaluation. It categorizes current explanation methods for deep neural networks according to their number of inputs. Additionally, it introduces the principles and evaluation methods for common interpretability algorithms. Future research directions and applications of interpretable neural networks are summarized. The challenges facing interpretable neural networks are outlined, along with potential solutions to these challenges.

Winter wheat identification in China through FY-3D NDVI&EVI satellite data and DNN Fusion

ABSTRACT Medium Resolution Spectral Imager-II (MERSI-II) on FengYun 3D (FY-3D) meteorological satellite is a visible and infrared spectral imaging instrument comparable with Moderate Resolution Imaging Spectroradiometer (MODIS), which provides ample opportunities for regional crop mapping. However, current research may have overlooked the potential of combining FY-3D MERSI-II data with deep neural network (DNN) algorithms for winter wheat identification. This research utilizes a DNN method to train a nonlinear model with the synthetic normalized difference vegetation index (NDVI) and enhanced vegetation index (EVI) at a 250 m resolution from FY-3D MERSI-II as the eigenvalues, enabling to accurately identify spatial distribution of winter wheat in China’s main production regions and filling the gap in the application of FY-3D MERSI-II data. The results indicate that the winter wheat identification with the combination of NDVI and EVI achieved an accuracy of 95.32%, with a Kappa coefficient of 0.87 and a determination coefficient (R 2 ) of 0.73, improving the overall accuracy by 1.27% and 1.07% compared to models trained solely using NDVI or EVI, respectively. In terms of spatial distribution, the overlap rate with the winter wheat dataset of China from MODIS at a 500 m resolution exceeded 95%, with an area bias of 2.67%. This study indicates that the medium resolution FY-3D MERSI-II remote sensing dataset combined with the DNN algorithm can accurately identify crop information over large areas.

Deep neural network-based molecular dynamics simulations for AlxGa1-xN alloys and their thermal properties

Efficient heat dissipation is crucial for the performance and lifetime of high electron mobility transistors (HEMTs). The thermal conductivity of materials and interfacial thermal conductance (ITC) play significant roles in their heat dissipation. To predict the thermal properties of AlxGa1-xN and the ITC of GaN/AlxGa1-xN in HEMTs, a dataset with first-principles accuracy was constructed using concurrent learning method and trained to obtain an interatomic potential employing deep neural networks (DNN) method. Using obtained DNN interatomic potential, equilibrium molecular dynamics simulations were employed to calculate the thermal conductivity of AlxGa1-xN, which showed excellent consistent with experimental results. Additionally, the phonon density of states of AlxGa1-xN and the ITC of GaN/AlxGa1-xN were calculated. Our study revealed a decrease in the ITC of GaN/AlxGa1-xN with increasing x, and the insertion of 1nm-thick AlN at the interface significantly reduced the ITC. This work provided a high-fidelity DNN potential for molecular dynamics simulations of AlxGa1-xN, offering valuable guidance for exploring the thermal transport of complex alloy and heterostructure.&#xD.

How can geostatistics help us understand deep learning? An exploratory study in SAR-based aircraft detection

Deep Neural Networks (DNNs) have garnered significant attention across various research domains due to their impressive performance, particularly Convolutional Neural Networks (CNNs), known for their exceptional accuracy in image processing tasks. However, the opaque nature of DNNs has raised concerns about their trustworthiness, as users often cannot understand how the model arrives at its predictions or decisions. This lack of transparency is particularly problematic in critical fields such as healthcare, finance, and law, where the stakes are high. Consequently, there has been a surge in the development of explanation methods for DNNs. Typically, the effectiveness of these methods is assessed subjectively via human observation on the heatmaps or attribution maps generated by eXplanation AI (XAI) methods. In this paper, a novel GeoStatistics Explainable Artificial Intelligence (GSEAI) framework is proposed, which integrates spatial pattern analysis from Geostatistics with XAI algorithms to assess and compare XAI understandability. Global and local Moran’s I indices, commonly used to assess the spatial autocorrelation of geographic data, assist in comprehending the spatial distribution patterns of attribution maps produced by the XAI method, through measuring the levels of aggregation or dispersion. Interpreting and analyzing attribution maps by Moran’s I scattergram and LISA clustering maps provide an accurate global objective quantitative assessment of the spatial distribution of feature attribution and achieves a more understandable local interpretation. In this paper, we conduct experiments on aircraft detection in SAR images based on the widely used YOLOv5 network, and evaluate four mainstream XAI methods quantitatively and qualitatively. By using GSEAI to perform explanation analysis of the given DNN, we could gain more insights about the behavior of the network, to enhance the trustworthiness of DNN applications. To the best of our knowledge, this is the first time XAI has been integrated with geostatistical algorithms in SAR domain knowledge, which expands the analytical approaches of XAI and also promotes the development of XAI within SAR image analytics.

A deep equivariant neural network approach for efficient hybrid density functional calculations

Hybrid density functional calculations are essential for accurate description of electronic structure, yet their widespread use is restricted by the substantial computational cost. Here we develop DeepH-hybrid, a deep equivariant neural network method for learning the hybrid-functional Hamiltonian as a function of material structure, which circumvents the time-consuming self-consistent field iterations and enables the study of large-scale materials with hybrid-functional accuracy. Our extensive experiments demonstrate good reliability as well as effective transferability and efficiency of the method. As a notable application, DeepH-hybrid is applied to study large-supercell Moiré-twisted materials, offering the first case study on how the inclusion of exact exchange affects flat bands in magic-angle twisted bilayer graphene. The work generalizes deep-learning electronic structure methods to beyond conventional density functional theory, facilitating the development of deep-learning-based ab initio methods.

A Bi‐level stacked LSTM‐DNN‐based decoder network for AGC dispatch under regulation market framework in presence of VPP and EV aggregators

AbstractThe consideration of mileage settlement in the frequency regulation market has encouraged fast‐acting units, such as converter‐interfaced generators (CIG) and electric vehicle stations, to actively participate in load‐generation balancing through automatic generation control (AGC). Conventional frequency regulation faces challenges in coping with the growing variability of CIGs and also lacks effective incentives for rapid‐responding units. In this context, a bi‐level AGC dispatch approach based on a stacked long short‐term memory (LSTM)‐deep neural network (DNN)‐based decoder framework is proposed for a power system comprising diverse CIGs forming a virtual power plant and electric vehicle aggregators. The proposed decoder network is comprised of stacked LSTM and DNN, wherein the cascaded LSTM layers are introduced to accurately capture temporal information from time series input. The inclusion of a dropout mechanism further enhances the model’s generalisability in unforeseen environments. The proposed dispatch framework uses mileage‐based compensation criteria to optimally allocate instructions among various participating units with differing regulation characteristics. The performance of the proposed method is analysed by considering packet loss, delay, unexpected generation failure, and denial of service attacks. The evaluation of the proposed approach reveals its superior performance compared to proportionality, particle swarm optimisation, decision tree, and DNN methods.

Predicting SARS-CoV-2 infection among hemodialysis patients using deep neural network methods

COVID-19 has a higher rate of morbidity and mortality among dialysis patients than the general population. Identifying infected patients early with the support of predictive models helps dialysis centers implement concerted procedures (e.g., temperature screenings, universal masking, isolation treatments) to control the spread of SARS-CoV-2 and mitigate outbreaks. We collect data from multiple sources, including demographics, clinical, treatment, laboratory, vaccination, socioeconomic status, and COVID-19 surveillance. Previous early prediction models, such as logistic regression, SVM, and XGBoost, require sophisticated feature engineering and need improved prediction performance. We create deep learning models, including Recurrent Neural Networks (RNN) and Convolutional Neural Networks (CNN), to predict SARS-CoV-2 infections during incubation. Our study shows deep learning models with minimal feature engineering can identify those infected patients more accurately than previously built models. Our Long Short-Term Memory (LSTM) model consistently performed well, with an AUC exceeding 0.80, peaking at 0.91 in August 2021. The CNN model also demonstrated strong results with an AUC above 0.75. Both models outperformed previous best XGBoost models by over 0.10 in AUC. Prediction accuracy declined as the pandemic evolved, dropping to approximately 0.75 between September 2021 and January 2022. Maintaining a 20% false positive rate, our LSTM and CNN models identified 66% and 64% of positive cases among patients, significantly outperforming XGBoost models at 42%. We also identify key features for dialysis patients by calculating the gradient of the output with respect to the input features. By closely monitoring these factors, dialysis patients can receive earlier diagnoses and care, leading to less severe outcomes. Our research highlights the effectiveness of deep neural networks in analyzing longitudinal data, especially in predicting COVID-19 infections during the crucial incubation period. These deep network approaches surpass traditional methods relying on aggregated variable means, significantly improving the accurate identification of SARS-CoV-2 infections.

A generalized Lorenz system-based initialization method for deep neural networks

Deep neural networks (DNNs) are a powerful tool for solving complex problems. The effectiveness of DNNs largely depends on the initialization technique used. This research develops a new initialization method for DNNs that uses chaotic sequences from the generalized Lorenz system to improve their performance. The proposed method, termed Generalized Lorenz Initialization (GLI), has been compared with two established initialization methods (Kaiming and Xavier) across four different DNN architectures: Informer, Neural Basis Expansion Analysis for Interpretable Time Series, Long Short-Term Memory, and NeuRewriter. The performance of these methods has been evaluated on seven time series forecasting datasets and one combinatorial optimization dataset. Results show that the GLI method improved forecasting accuracy by up to 86.47% compared to the Kaiming method and 88.86% compared to the Xavier method across all time series datasets. For the combinatorial optimization task, the GLI method reduced computational time by up to 9.24% with the better solution quality. These indicate the superiority of the GLI method over the two representative initialization methods for different DNN architectures across different problem domains.

Modeling CO2 loading capacity of triethanolamine (TEA) aqueous solutions via a deep learning approach

Capturing carbon dioxide (CO2) from natural gas is essential for reducing CO2 emissions as a greenhouse gas as well as increasing the gas heating value. Absorption of CO2 by amine solutions is a method that has seen a widespread use as a CO2 collection technique. In this research, advanced ML approaches are utilized to simulate the CO2 loading capacity of triethanolamine (TEA) aqueous solutions as a function of system temperature, CO2 partial pressure, and amine concentration in the aqueous phase. Deep neural network (DNN), Gaussian process regressor (GPR), deep belief network (DBN) and bagging regressor (BR) are the models that have been developed. The DNN model with the coefficient of determination (R2) of 0.9990 as well as the root mean square error (RMSE) of 0.0073 outperformed other models in terms of accuracy and validity. The R2 values of 0.9911, 0.9861, and 0.9745 for DBN, BR, and GPR, accordingly, showed that other models could likewise perform with high accuracy. Moreover, trend analysis of the results confirmed that the DNN method can correctly estimate the behavior of CO2 loading with variations in the input parameters. Furthermore, a sensitivity analysis showed that temperature has a decreasing effect on the CO2 loading, while amine concentration and CO2 partial pressure exhibited to have an incremental effect. Herein, the temperature had the greatest influence on the amount of CO2 loading. The final stage of the research was the leverage approach, which stated that about 99% of the data are statistically valid and the DNN model is reliable to be replaced by experimental approaches in the prediction of CO2 absorption into certain type of solutions.

Multi-scale attention in attention neural network for single image deblurring

Image deblurring, which eliminates blurring artifacts to recover details from a given input image, represents an important task for the computer vision field. Recently, the attention mechanism with deep neural networks (DNN) demonstrates promising performance of image deblurring. However, they have difficulty learning complex blurry and sharp relationships through a balance of spatial detail and high-level contextualized information. Moreover, most existing attention-based DNN methods fail to selectively exploit the information from attention and non-attention branches. To address these challenges, we propose a new approach called Multi-Scale Attention in Attention (MSAiA) for image deblurring. MSAiA incorporates dynamic weight generation by leveraging the joint dependencies of channel and spatial information, allowing for adaptive changes to the weight values in attention and non-attention branches. In contrast to existing attention mechanisms that primarily consider channel or spatial dependencies and do not adequately utilize the information from attention and non-attention branches, our proposed AiA design combines channel-spatial attention. This attention mechanism effectively utilizes the dependencies between channel-spatial information to allocate weight values for attention and non-attention branches, enabling the full utilization of information from both branches. Consequently, the attention branch can more effectively incorporate useful information, while the non-attention branch avoids less useful information. Additionally, we employ a novel multi-scale neural network that aims to learn the relationships between blurring artifacts and the original sharp image by further exploiting multi-scale information. The experimental results prove that the proposed MSAiA achieves superior deblurring performance compared with the state-of-the-art methods.

Development of Speech Recognition in Wireless Mobile Networks for An Intelligent Learning System in Language Education

Communication has been crucial to human existence, society, and globalization for millennia. Speech Recognition (SR) technologies include biometric evaluation, security, safety, medical care, and smart cities. Most research has primarily focused on English; others neglect other lower-asset dialects like Uzbek, neglecting its research unaddressed. This study examines the efficacy of peer and ASR response in wireless mobile networks-assisted pronouncing training. This study proposes a Deep Neural Network (DNN) and Hidden Markov Method (HMM) based ASR model to develop a voice recognition system utilizing a combination of Connected Time-based Categorization (CTC)-attending networks for the Uzbek words and their variants. The suggested method diminishes training duration and enhances SR precision by efficiently employing the CTC goal function in attentiveness modeling. The research assessed the results of both linguistic experts and native speakers on the Uzbek database, which was compiled for this research. The data were gathered through a pronunciation assessment and a discussion. The participant was further instructed in the classroom. Test outcomes indicate that the suggested method attained a word error rate of 13.1%, utilizing 210 hours of records as a learning dataset for the Uzbek dialect. The proposed technique can significantly enhance students' pronunciation qualities. It might inspire pupils to participate in pronunciation learning.

Identification of Effective Factors and Prediction of Ischemic Heart Disease Using Machine Learning Methods and Data from the Yazd Health Study (YaHS) "

Introduction: Ischemic heart diseases are one of the most common diseases that cause high mortality worldwide. This article has identified various factors affecting heart disease and identified susceptible people using various machine learning methods. Methods: The current research was conducted on the Yazd Health Study (YaHS) database. YaHS was conducted on adults aged 20-70 years who were residents of Yazd Greater Area and collected information on the health and various diseases of nearly 10,000 people in the form of a questionnaire with 300 different questions. In this research, by using the correlation of questions with heart disease, the most important factors of heart disease have been identified. By using the identified factors and machine learning algorithms, susceptible people with heart disease have been identified. Results: The results of the evaluations have shown that factors such as age, family history of heart disease, blood pressure, diabetes, blood cholesterol, stress, anxiety, depression, quality of life, quality of sleep, physical activity, smoking, and diet have an effect on heart disease. Likewise, among the different machine learning methods, the nearest neighbor method, the deep neural network method, and the multi-layer perceptron method with a recall criterion of 99.94%, 99.88%, and 99.11% have performed the best in the identifying sick people, respectively. Conclusion: According to the findings of the research, it can be understood that by controlling factors such as blood pressure, diabetes, blood cholesterol, stress, anxiety, and depression, changing factors such as quality of life, sleep status, physical activity, and eating patterns of people and quitting smoking, it is possible to move towards improving the health of society. On the other hand, the identification of people prone to heart disease using machine learning methods is done faster and at a lower cost than the traditional methods that are done by referring to medical centers and doctors and performing various tests.

Wind Field Reconstruction Method Using Incomplete Wind Data Based on Vision Mamba Decoder Network

Accurate meteorological information is crucial for the safety of civil aviation flights. Complete wind field information is particularly helpful for planning flight routes. To address the challenge of accurately reconstructing wind fields, this paper introduces a deep learning neural network method based on the Vision Mamba Decoder. The goal of the method is to reconstruct the original complete wind field from incomplete wind data distributed along air routes. This paper proposes improvements to the Vision Mamba model to fit our mission, showing that the developed model can accurately reconstruct the complete wind field. The experimental results demonstrate a mean absolute error (MAE) of wind speed of approximately 1.83 m/s, a mean relative error (MRE) of around 7.87%, an R-square value of about 0.92, and an MAE of wind direction of 5.78 degrees.

EAV: EEG-Audio-Video Dataset for Emotion Recognition in Conversational Contexts

Understanding emotional states is pivotal for the development of next-generation human-machine interfaces. Human behaviors in social interactions have resulted in psycho-physiological processes influenced by perceptual inputs. Therefore, efforts to comprehend brain functions and human behavior could potentially catalyze the development of AI models with human-like attributes. In this study, we introduce a multimodal emotion dataset comprising data from 30-channel electroencephalography (EEG), audio, and video recordings from 42 participants. Each participant engaged in a cue-based conversation scenario, eliciting five distinct emotions: neutral, anger, happiness, sadness, and calmness. Throughout the experiment, each participant contributed 200 interactions, which encompassed both listening and speaking. This resulted in a cumulative total of 8,400 interactions across all participants. We evaluated the baseline performance of emotion recognition for each modality using established deep neural network (DNN) methods. The Emotion in EEG-Audio-Visual (EAV) dataset represents the first public dataset to incorporate three primary modalities for emotion recognition within a conversational context. We anticipate that this dataset will make significant contributions to the modeling of the human emotional process, encompassing both fundamental neuroscience and machine learning viewpoints.

Integrated study of prediction and optimization performance of PBI-HTPEM fuel cell using deep learning, machine learning and statistical correlation

This paper uses 3D modeling and artificial intelligence methods to predict, and find optimal point in high-temperature proton exchange membrane (HTPEM) fuel cells. The main objective is to obtain maximum power and current density at the optimum node of the HTPEM fuel cell. The response surface method (RSM) is used to prevent excessive duplication and ensure adequate data coverage for determining input parameters. Also, for the first time, the correlation presented was compared with AI-based metaheuristic optimization methods i.e., including support vector regression (SVR), Gaussian process regression (GPR), and deep neural networks (DNN) with a dropout layer, alongside metaheuristic algorithms such as whale optimization algorithm (WOA), Grasshopper optimization algorithm (GOA), firefly algorithm (FF), and the genetic algorithm (GA). The results show that SVR, GPR, and DNN methods have excellent performance, with mean absolute percentage error (MAPE) of 0.81 % for DNN, 0.83 % for SVR, and 2.24 % for GPR. Most optimization algorithms exhibit errors below 8 %. The DNN-GOA, SVR-WOA, SVR-GA, and GPR-GOA algorithms have the lowest errors among them. Correlations have a lower computational cost for obtaining maximum power and current density at the optimum node compared to optimization algorithms, with a relative error of less than 6 % in most cases.

Machine learning optimization and 4E analysis of a CCHP system integrated into a greenhouse system for carbon dioxide capturing

The world has seen an increase in the popularity of renewable-based energy systems. However, because of their erratic nature, fossil fuels cannot be entirely phased out. Utilizing carbon dioxide in greenhouses close to power plants and generating extra power electricity from dissipated thermal energy are appealing strategies that cause us to have a cleaner environment and affordable power electricity. To exploit the high enthalpy of expelled gases from a gas turbine, the gas turbine was coupled with an absorption refrigerant cycle and an organic Rankine cycle and analyzed. Additionally, a deep artificial neural network method was used to calculate all functions in a rapid optimization and accurate 4E analysis. The carbon dioxide emissions subsided from 0.9183 to 0.41 kg CO2/s, and the optimization time improved 257 times after using the trained machine learning model. An adsorption technique for CO2 separation is used for the proposed system because of its lower energy consumption. The maximum exergy and energy efficiencies were attained at 36.71 % and 49.2 %, respectively, and parametric studies are being assessed. Due to increases in harvests and efficiencies, the net annual interest has increased by 21.8 %, up to 22.5 million dollars.

Digitalization of Analysis of a Concrete Block Layer Using Machine Learning as a Sustainable Approach

The concrete block pavement (CBP) system has a surface layer consisting of concrete block pavers and joint sand over a bedding sand layer. The non-homogeneous nature of the surface course of CBP, along with different laying patterns and shapes of block pavers, makes the analysis of CBP cumbersome. In this study, the surface course of CBP was modeled based on the slab action of the block pavers and joint sand, which are connected together in full contact. Four different laying patterns, including herringbone, stretcher, parquet, and square, were modeled using a finite element model. The elastic moduli of the block pavers varied from 2500 MPa to 45,000 MPa, with thicknesses ranging from 60 mm to 120 mm. As a result, modeling of CBP based on slab action can be considered a realistic strategy. In addition, a dataset was created based on quantitative inputs, e.g., elastic modulus and thickness of the block pavers, and qualitative input, i.e., block laying patterns. The approaches of machine learning adopted were support vector regression, Gaussian process regression, single-layer and deep artificial neural networks, and least squares boosting to implement prediction approach based on input and output. The analyses of statistical accuracy of all five machine learning methods showed high accuracy; however, the Gaussian process and deep artificial neural network methods resulted in the most accurate outputs and are recommended for further studies. Based on the machine learning models, digitalization is achieved through the development of simple, user-friendly software for electronic devices in order to perform a preliminary analysis of different laying patterns of CBP. Such a platform may result in less laboratory work and boosts the level of sustainability in concrete block pavement technology.