Neural Network Architecture Research Articles

Enhancing accuracy in X-ray radiation-based multiphase flow meters: Integration of grey wolf optimization and MLP neural networks

This research investigates the development of an advanced predictive model aimed at accurately determining the volumetric percentages of water, oil, and gas within oil pipeline systems. Utilizing an innovative approach that incorporates an X-ray source alongside two sodium iodide detectors, the study leverages the Monte Carlo N-Particle (MCNP) simulation code to model the behavior of three-phase fluids under varied conditions. The model meticulously simulates various volumetric configurations of water, oil, and gas, resulting in a comprehensive dataset that provides key spectral information. The initial phase involved the extraction of ten temporal and frequency-related features from each detector, culminating in a pool of twenty features. The analytical process then applied the Grey Wolf Optimization (GWO) algorithm to select the most indicative features for predictive modeling. Out of the initial set, seven features—short-time energy, frequency deviation, relative spectral density, spectral margin, main peak position, spectral coefficient, and frequency intensity—were identified as critical for enhancing model accuracy. These features were subsequently fed into a meticulously structured multilayer perceptron (MLP) neural network. This network, designed with two hidden layers containing 20 and 10 neurons, respectively, demonstrated exceptional capability, achieving a root mean square error (RMSE) of less than 0.06 in the prediction of oil and gas volumetric percentages. The study emphasizes the significant impact of integrating refined feature selection techniques and robust neural network architectures on the precision and reliability of volumetric predictions in multiphase flow systems within oil pipelines. This approach not only enhances predictive accuracy but also contributes to more efficient resource management and operational planning in the oil and gas industry.

Use of ResNets for HLB Disease Detection on Orange Leaves Using Terrestrial Multispectral Images

Abstract. Huanglongbing (HLB) is a bacterial disease transmitted by different vectors of sap-sucking insects. It affects all crops of citrus trees, decreasing the values of those fruits in the market and eventually the decay of orchards. In Brazil, the world's leading orange producer, citriculture faces severe issues with HLB and substantial economic loss. Technical means of scanning the orchards with high-throughput becomes essential for the sustainability of this industry. In this study, we propose to investigate an operational strategy consisting of scanning large portions of foliage (the canopy of one tree or more) in which there can few early foliage symptoms. It is proposed to investigate deep learning tools to solve this complex binary classification problem. The study is based on a dataset comprising 1,297 terrestrial multispectral (14 channels) images captured at high spatial resolution in a commercial orange orchard in Brazil. It is proposed to adapt and retrain standard neural network architectures, namely ResNets18 and ResNets34, to process such images. Our analysis reveals promising results, with both models demonstrating convergence and achieving stable performance. Notably, ResNet18 outperformed ResNet34, achieving an accuracy of 76.45% compared to 66.79% from ResNet34. These findings suggest that deep neural network methods can effectively manage non-radiometrically calibrated data and accurately distinguish images with HLB symptoms from healthy plants . However, with reduced datasets and limited possibilities for transfer learning and fine-tuning, it seems that only reasonable sized networks can be trained. Thus, more advanced state-of-the-art tools of the are still challenging to deploy for agricultural multi- or hyperspectral data.

Multi-source partial discharge pattern recognition in GIS based on Grabcut-MCNN

Partial discharge (PD) surveillance constitutes a pivotal methodology for diagnosing insulation failures in electrical equipment. Enhancing comprehensively the precision of identifying PD anomalies in Gas Insulated Switchgear (GIS) is of paramount significance for ensuring the steady functioning of power grids. This study introduces a novel framework that integrates Phase-Resolved PD Graph Segmentation (PRPD-Grabcut) with a tailored MobileNets-based Convolutional Neural Network (MCNN) to classify GIS-related PD issues. Leveraging image segmentation via PRPD-Grabcut, crucial features are extracted from PRPD diagrams, which then facilitate the construction of the MCNN model. This model employs depth-wise separable convolutions alongside inverted residual architectures to tackle the vanishing gradient dilemma inherent in Deep Convolutional Neural Networks (DCNNs) during GIS PD pattern discernment. Upon the model's subsequent training and validation, empirical evidence illustrates that the PRPD-Grabcut-MCNN hybrid significantly alleviates the computational load and storage requisites of the model, concurrently enhancing the recognition precision and expediting the training process of the neural network. Relative to diverse established lightweight neural network architectures, MCNN manifests superior performance in terms of recognition accuracy, reduced cross-entropy loss, and expedited training duration.

Nested Neural Networks: A Novel Approach to Flexible and Deep Learning Architectures

It’s easy to get lost in the complexities of sentences like, “Never will I have been in a place whose experiences will have been teaching me how amazing life really is till the beginning of every one of those marvelous whole new advenchallenges for more than 20 years of life by the end of my military service.” These kinds of nested and layered statements are not just the domain of advanced grammar—they have a parallel in cutting-edge technology as well. This paper introduces a novel neural network architecture, termed Nested Neural Networks (NNN), that incorporates nested layers within a more complex overarching structure. Much like the layered conditions in Future Perfect and Future Perfect Progressive tenses, NNNs utilize ‘nested’ conditions to bring flexibility and depth to data processing. This structure captures complex relationships, achieving high performance, computational efficiency, and generalization across tasks, much as understanding the nested clauses of a sentence can reveal deeper meanings. Our experiments on benchmark datasets demonstrate NNNs’ ability to handle intricate data patterns with enhanced memory efficiency and training adaptability, often outperforming traditional models. This fascinating intersection between language and machine learning showcases the powerful potential of structured complexity, whether in communication or computation.

Reconstruction of blood flow velocity with deep learning information fusion from spectral ct projections and vessel geometry

In this work, we investigate a new deep learning reconstruction method of blood flow velocity within deformed vessels from contrast enhanced X-ray projections and vessel geometry. The principle of the method is to perform linear or nonlinear dimension reductions on the Radon projections and on the mesh of the vessel. These low dimensional projections are then fused to obtain the velocity field in the vessel. The accuracy of the reconstruction method is proved using various neural network architectures with realistic unsteady blood flows. The approach leverages the vessel geometry information and outperforms the simple PCA-net.

Deep Learning and Handcrafted Features for Thyroid Nodule Classification

ABSTRACTIn this research, we present a refined image‐based computer‐aided diagnosis (CAD) system for thyroid cancer detection using ultrasound imagery. This system integrates a specialized convolutional neural network (CNN) architecture designed to address the unique aspects of thyroid image datasets. Additionally, it incorporates a novel statistical model that utilizes a two‐dimensional random coefficient autoregressive (2D‐RCA) method to precisely analyze the textural characteristics of thyroid images, thereby capturing essential texture‐related information. The classification framework relies on a composite feature vector that combines deep learning features from the CNN and handcrafted features from the 2D‐RCA model, processed through a support vector machine (SVM) algorithm. Our evaluation methodology is structured in three phases: initial assessment of the 2D‐RCA features, analysis of the CNN‐derived features, and a final evaluation of their combined effect on classification performance. Comparative analyses with well‐known networks such as VGG16, VGG19, ResNet50, and AlexNet highlight the superior performance of our approach. The outcomes indicate a significant enhancement in diagnostic accuracy, achieving a classification accuracy of 97.2%, a sensitivity of 84.42%, and a specificity of 95.23%. These results not only demonstrate a notable advancement in the classification of thyroid nodules but also establish a new standard in the efficiency of CAD systems.

Leveraging Edge Computing and Deep Learning for the Real-Time Identification of Bean Plant Pathologies

Beans are essential crops globally, standing out as one of the most consumed and nourishing legumes, thereby playing a significant role in human nutrition and food security. Their cultivation faces several challenges, such as pests, diseases, unpredictable weather patterns, and soil erosion. Of these challenges, diseases are recognized as a key challenge, resulting in a decline in both yield quality and quantity, and inflicting substantial financial losses on farmers.This work proposes a deep learning-based approach for precise in-field identification of diseases in bean plants. We evaluate image classification and object detection models using state-of-the-art Convolutional Neural Network (CNN) architectures to identify Angular Leaf Spot (ALS) and bean rust diseases, key bean diseases in Uganda and the region in general, from smartphone images of bean leaves collected in various districts of Uganda.The dataset employed to train these models is the Makerere University beans image dataset, comprising 15,335 images categorized into three (3) classes (ALS, bean rust, and healthy). To improve in-field performance, the dataset was expanded to include an additional class (unknown class) consisting of a diverse collection of 2,800 images to account for images unrelated to the three (3) predefined classes. Adversarial training was further employed to enhance model robustness in identifying the target classes. In addition, two (2) Out-of-Distribution (ODD) detection techniques, i.e., confidence thresholding and training with an auxiliary class (unknown class), were utilized to handle inputs unrelated to bean leaves.Our results show that our custom CNN, BeanWatchNet, achieved 90% accuracy when tested on unseen data for the classification of the three (3) target classes, i.e., ALS, bean rust and healthy. EfficientNet v2 B0 and BeanWatchNet demonstrated superior performance for the four-class (with unknown class) image classification task, achieving 91% and 90% accuracy, respectively, when evaluated on the test dataset. YOLO v8 exhibited superior performance for the object detection models, attaining mAP@50 of 87.6. The custom CNN model and YOLO v8 model were quantized and deployed across two (2) edge platforms: a smartphone (through a mobile application) and a Raspberry Pi 4B to facilitate in-field disease detection. The benchmarking code and best models are publicly available on GitHub.1

Multivariate probabilistic forecasting of electricity prices with trading applications

This study extends recently introduced neural networks approach, based on a regularized distributional multilayer perceptron (DMLP) technique for a multivariate case electricity price forecasting. The performance of a fully connected architecture and a LSTM architecture of neural networks are tested. Different from previous studies we incorporate dependence between multiple exchanges (EPEX and Nord Pool). The empirical data application analyzes two auctions in the day-ahead electricity market for the United Kingdom market. Along with statistical evaluation of probabilistic forecasts we develop a flexible bidding strategy based on risk-adjusted investor utility function. The trading application leverages the differences of the two exchanges by having long/short positions in both. Our findings demonstrate while DMLP shows similar performance compared to the benchmarks, the algorithm is considerably less computationally costly. LASSO Quantile Regression is better in terms if statistical evaluation of distributional fit, while DMLP outperforms in terms of Sharpe ration (by 18%) in the trading application.

A novel Taguchi-based approach for optimizing neural network architectures: application to elastic short fiber composites

This study presents an innovative application of the Taguchi design of experiment method to optimize the structure of an Artificial Neural Network (ANN) model for the prediction of elastic properties of short fiber reinforced composites. The main goal is to minimize the computational effort required for hyperparameter optimization while enhancing the prediction accuracy. By utilizing a robust experimental design framework, the structure of an ANN model is optimized. This approach involves identifying a combination of hyperparameters that provides optimal predictive accuracy with the fewest algorithmic runs, thereby significantly reducing the required computational effort. The results suggested that the Taguchi-based developed ANN model with three hidden layers, 20 neurons in each hidden layer, elu activation function, Adam optimizer, and a learning rate of 0.001 can predict the anisotropic elastic properties of short fiber reinforced composites with a prediction accuracy of 97.71%. Then, external validation of the proposed ANN model was done using experimental data, and differences of less than 10% were obtained, indicating an appropriate predictive performance of the proposed ANN algorithm. Our findings demonstrate that the Taguchi method not only streamlines the hyperparameter tuning process but also substantially improves the algorithm's performance. These results highlight the potential of the Taguchi method as a powerful tool for optimizing machine learning algorithms, especially in scenarios where computational resources are limited. The implications of this study are far-reaching, offering insights for future research in the optimization of different algorithms for improved accuracies and computational efficiencies.

A novel automated neural network architecture search method of air target intent recognition

Modern air battlefield operations are characterized by flexibility and change, and the battlefield evolves rapidly and intricately. However, traditional air target intent recognition methods, which mainly rely on manually designed neural network models, find it difficult to maintain sustained and excellent performance in such a complex and changing environment. To address the problem of the adaptability of neural network models in complex environments, we propose a lightweight Transformer model (TransATIR) with a strong adaptive adjustment capability, based on the characteristics of air target intent recognition and the neural network architecture search technique. After conducting extensive experiments, it has been proved that TransATIR can efficiently extract the deep feature information from battlefield situation data by utilizing the neural architecture search algorithm, in order to quickly and accurately identify the real intention of the target. The experimental results indicate that TransATIR significantly improves recognition accuracy compared to the existing state-of-the-art methods, and also effectively reduces the computational complexity of the model.

From One-dimensional to Multidimensional Map Neural Networks

In the contemporary landscape, artificial neural networks transcend their traditional role of function approximation and have found diverse applications in fields such as image classification, machine translation, speech recognition, and natural language processing. In some datasets, traditional architectures exhibit low test and training accuracy, coupled with high loss and prolonged training times. This study aims to introduce innovative neural network architectures that outperform conventional models. The research presents a novel framework integrating one-dimensional and multidimensional map neural networks, consisting of three distinct architectures: 1D-Map, 2D-Map, and 3D-Map. A systematic performance comparison with traditional models is conducted following the implementation of these architectures. The evaluation spans four datasets, encompassing the domains of heat treatment of electroless Ni-P nano coatings, letter recognition, combined cycle power plants, and Seoul bike sharing demand. In the heat treatment dataset, the proposed 3D-Map architecture reached 0.055 higher average test accuracy than traditional MLP architecture. In the letter recognition dataset, the test accuracy of 3D-Map architecture was 0.0523 higher than the test accuracy of the LSTM architecture. In the combined cycle power plant dataset, the test accuracy of the 3D-Map architecture was 0.0612 more than the test accuracy of the MLP architecture. In the Seoul bike-sharing demand dataset, the test accuracy of the 2D-Map architecture was 0.0696 higher than the test accuracy of the LSTM architecture. The study's findings underscore the consistently superior performance of the proposed architectures compared to their traditional counterparts.

Probabilistic oil price forecasting with a variational mode decomposition-gated recurrent unit model incorporating pinball loss

Prediction methods have garnered significant attention in intelligent decision-making. Most existing approaches to predicting crude oil prices prioritize accuracy and stability while providing precise prediction intervals (PIs) that can offer valuable insights. Thus far, we introduce a novel hybrid model to forecast future crude oil prices. Our approach leverages the variational mode decomposition (VMD) to simplify the complexity of the original time series, yielding a set of subseries. These subseries are then modeled using a deep neural network architecture called a gated recurrent unit (GRU). To address the prediction uncertainty, we employed the pinball loss function rather than the mean square error (MSE) to guide the proposed VMD-GRU. This adaptation extends the traditional GRU-based point forecasting to probabilistic forecasting by estimating quantiles. We evaluated our proposed model on a well-established crude oil price series by conducting both single- and multi-step-ahead forecasting analyses. Our findings underscore the efficacy of the combined model, demonstrating its superior predictive performance compared to benchmark models.

Interactions of localized wave and dynamics analysis in the new generalized stochastic fractional potential-KdV equation.

In this paper, we investigate the new generalized stochastic fractional potential-Korteweg-de Vries equation, which describes nonlinear optical solitons and photon propagation in circuits and multicomponent plasmas. Inspired by Kolmogorov-Arnold network and our earlier work, we enhance the improved bilinear neural network method by using a large number of activation functions instead of neurons. This method incorporates the concept of simulating more complicated activation functions with fewer parameters, with more diverse activation functions to generate more complex and rare analytical solutions. On this basis, constraints are introduced into the method, reducing a significant amount of computational workload. We also construct neural network architectures, such as "2-3-1," "2-2-3-1," "2-3-3-1," and "2-3-2-1" using this method. Maple software is employed to obtain many exact analytical solutions by selecting appropriate parameters, such as the superposition of double-period lump solutions, lump-rogue wave solutions, and three interaction solutions. The results show that these solutions exhibit more complex waveforms than those obtained by conventional methods, which is of great significance for the electrical systems and multicomponent fluids to which the equation is applied. This novel method shows significant advantages when applied to fractional-order equations and is expected to be increasingly widely used in the study of nonlinear partial differential equations.

Beyond breathalyzers: AI-powered speech analysis for alcohol intoxication detection

Detecting potential alcohol inebriation or intoxication status holds paramount significance for social prevention and security. Beyond its association with long-term health effects, alcohol consumption can lead to immediate consequences, including reduced control over one’s actions, with traffic fatalities representing one of the most tragic outcomes.This study leveraged the Alcohol Language corpus, involving 162 subjects recorded both in sober and inebriated states. Participants provided 60 speech samples while sober and 30 when intoxicated, all within a realistic car setting using head-mounted microphones. Our research endeavors encompassed comprehensive stratified statistical tests to examine the impact of alcohol consumption on speech production while uncovering the influence of covariates such as age, gender, and drinking habits.Additionally, we introduced a speaker-neutral machine learning algorithm, based on the Domain-Adversarial Neural Network architecture. This approach aimed to overcome challenges posed by individual differences that often complicate intoxicated speech analysis. Notably, our findings highlighted the effectiveness of features like the RASTA-filtered auditory spectrum. Nevertheless, the results from statistical tests emphasized the need for techniques that minimize inter-subject variability.As for the automatic classification, the proposed architecture exhibited promising results, yielding a classification accuracy slightly exceeding 70% on an independent test set. Although preliminary, our research demonstrates the potential for detecting alcohol-induced speech changes, benefiting societal well-being and security. It also underscores the importance of developing strategies that account for individual differences while harnessing the power of automatic models to effectively distinguish between sober and intoxicated individuals.

Multi-point temperature or humidity prediction for office building indoor environment based on CGC-BiLSTM deep neural network

The aim of this study is to predict the temperature or humidity changes at multiple relevant points in a building using a deep neural network architecture with multi-task learning to provide more reference information for the design and optimal operation of heating and ventilation systems. For this purpose, traditional multi-task prediction algorithm architecture is combined with Customized Gate Control and other neural networks to build deep neural network architectures for indoor environments with multi-point temperature or humidity prediction tasks. To test the prediction effectiveness of the architecture, a task of predicting temperature or humidity 24h in advance was designed on a real office building indoor environment dataset, and the prediction results were compared with other single-task and multi-task prediction models. Two experimental conditions were designed for this study, one using the complete training set and the other reducing the training set at a certain point. Through the final prediction results, it is found that the multi-task prediction architecture used in this paper shows better or nearly optimal results compared to other prediction models under both working conditions. This study provides some reference value for the application of multi-task prediction algorithms to the task of predicting indoor environments in buildings.

Point cloud segmentation neural network with same-type point cloud assistance

This paper proposes neural network architectures for point cloud segmentation, which leverage prior knowledge derived from same-type point clouds. The approach involves concurrent processing of two point clouds: a target point cloud necessitating segmentation and a labeled same-type point cloud. The labeled point cloud provides preliminary labeling information, assisting in segmenting the target point cloud. A feature combination module is proposed to identify and combine corresponding features across the point clouds. The module augments the feature representation of the target cloud and improves its capacity for object discrimination. Experiments on the ShapeNetPart and S3DIS datasets demonstrate that when integrated into classical network architectures, the proposed approach can achieve improved segmentation performance over the corresponding networks, significantly in some of them.

GPS-free autonomous navigation in cluttered tree rows with deep semantic segmentation

Segmentation-based autonomous navigation has recently been presented as an appealing approach to guiding robotic platforms through crop rows without requiring perfect GPS localization. Nevertheless, current techniques are restricted to situations where the distinct separation between the plants and the sky allows for the identification of the row’s center. However, tall, dense vegetation, such as high tree rows and orchards, is the primary cause of GPS signal blockage. In this study, we increase the overall robustness and adaptability of the control algorithm by extending the segmentation-based robotic guiding to those cases where canopies and branches occlude the sky and prevent the utilization of GPS and earlier approaches. An efficient Deep Neural Network architecture has been used to address semantic segmentation, performing the training with synthetic data only. Numerous vineyards and tree fields have undergone extensive testing in both simulation and real world to show the solution’s competitive benefits. The system achieved unseen results in orchards, with a Mean Average Error smaller than 9% of the maximum width of each row, improving state-of-the-art algorithms by disclosing new scenarios such as close canopy crops. The official code can be found at: https://github.com/PIC4SeR/SegMinNavigation.git.

PreAlgPro: Prediction of allergenic proteins with pre-trained protein language model and efficient neutral network

Allergy is a prevalent phenomenon, involving allergens such as nuts and milk. Avoiding exposure to allergens is the most effective preventive measure against allergic reactions. However, current homology-based methods for identifying allergenic proteins encounter challenges when dealing with non-homologous data. Traditional machine learning approaches rely on manually extracted features, which lack important protein functional characteristics, including evolutionary information. Consequently, there is still considerable room for improvement in existing methods. In this study, we present PreAlgPro, a method for identifying allergenic proteins based on pre-trained protein language models and deep learning techniques. Specifically, we employed the ProtT5 model to extract protein embedding features, replacing the manual feature extraction step. Furthermore, we devised an Attention-CNN neural network architecture to identify potential features that contribute to the classification of allergenic proteins. The performance of our model was evaluated on four independent test sets, and the experimental results demonstrate that PreAlgPro surpasses existing state-of-the-art methods. Additionally, we collected allergenic protein samples to validate the robustness of the model and conducted an analysis of model interpretability.

Latent-Energy-Based NNs: An interpretable Neural Network architecture for model-order reduction of nonlinear statics in solid mechanics

Latent-Energy-Based NNs: An interpretable Neural Network architecture for model-order reduction of nonlinear statics in solid mechanics

Deep learning-enhanced remote sensing-integrated crop modeling for rice yield prediction

This study introduces a novel crop modeling approach based on cutting-edge computational tools to advance crop production monitoring methodologies, and, thereby, tackle global food security issues. Our approach pioneers integrating deep learning and remote sensing with process-based crop models to enhance rice yield predictions while leveraging the strengths and weaknesses of each model. We developed and evaluated four models based on distinct deep neural network architectures: feed-forward neural network, long short-term memory (LSTM), gated recurrent units, and bidirectional LSTM. All the models demonstrated high predictive accuracies, with percent biases of 0.74–2.62 and Nash–Sutcliffe model efficiencies of 0.954–0.996; however, the LSTM performed best among the four models. Notably, the models' performances varied when applied to regional datasets that were not included in the training phase; this highlighted the critical need for diverse training data to enhance model robustness. This research marks a significant advancement in agricultural modeling by combining state-of-the-art computational techniques with established methodologies, setting a new standard for crop yield prediction.