Standard Neural Network Research Articles

Distance preserving machine learning for uncertainty aware accelerator capacitance predictions

Abstract Accurate uncertainty estimations are essential for producing reliable machine learning models, especially in safety-critical applications such as accelerator systems. Gaussian process models are generally regarded as the gold standard for this task; however, they can struggle with large, high-dimensional datasets. Combining deep neural networks with Gaussian process approximation techniques has shown promising results, but dimensionality reduction through standard deep neural network layers is not guaranteed to maintain the distance information necessary for Gaussian process models. We build on previous work by comparing the use of the singular value decomposition against a spectral-normalized dense layer as a feature extractor for a deep neural Gaussian process approximation model and apply it to a capacitance prediction problem for the High Voltage Converter Modulators in the Oak Ridge Spallation Neutron Source. Our model shows improved distance preservation and predicts in-distribution capacitance values with less than 1% error.

Fractional Gradient Optimized Explainable Convolutional Neural Network for Alzheimer's Disease Diagnosis

Alzheimer’s is one of the brain syndromes that steadily effects the brain memory. The early stage of Alzheimer’s disease referred as mild-cognitive-impairment (MCI) and the growth of Alzheimer’s is not certain in the patients with MCI. The premature Alzheimer’s diagnosis detection is crucial for the healthy brains functionality by avoiding memory loss. Different multi-neural network architectures have been suggested by the researchers for efficient and accurate Alzheimer’s disease (AD) detection. The absence of improved feature extraction mechanisms and unexplored efficient optimizers in complex benchmark architectures lead to an inefficient and inaccurate AD classification. Moreover, the standard Convolutional Neural Network (CNN)-based architectures for Alzheimer’s diagnosis lack interpretability in their predictions. An interpretable, simplified yet effective deep learning model is required for accurate classification of Alzheimer’s disease. In this study, a generalized fractional order-based CNN classifier with explainable artificial intelligence (XAI) capabilities is proposed for accurate, efficient and interpretable classification of AD diagnosis. The proposed study (a) classifies AD accurately by incorporating unexplored pooling technique with enhanced feature extraction mechanism (b) provides fractional order-based optimization approach for adaptive learning and fast convergence speed (c) suggests an interpretable method for proving the transparency of the model. The proposed model outclasses complex benchmark architectures with regard to accuracy using standard ADNI dataset. The suggested fractional order-based CNN classifier achieves an improved accuracy of 99% as compared to the state-of-the-art models.

Prediction method for 4D trajectory of large fixed wing UAV under multi-layer convolution and bidirectional gating model

Abstract With the increase of low-altitude flight tasks, the importance of research on low-altitude flight safety and efficient operation has gradually become prominent. Therefore, it is necessary to try to establish a low-altitude airspace flight path pre-planning method based on 4D trajectory prediction, which can effectively support the above research. In order to improve the integrity, accuracy and reliability of the long-range trajectory prediction results of large fixed wing UAV (LFW UAV), based on the standard neural network as the basic concept to form a fusion model of improved convolutional neural network (CNN) and bidirectional gated recurrent neural network (BIGRU). At the same time, the flight trajectory fitting preprocessing is added to build the completed multilayer cyclic convolutional improvement model (MCCI). The experimental results show that the MCCI model has obvious advantages in terms of prediction accuracy, deviation range and predictable duration when dealing with the LFW UAV 4D trajectory prediction problem, and better reflects the characteristics of 3D position and spatiotemporal elements. The comparative analysis shows that the error values of the MCCI model are smaller than those of the comparison prediction models in any dimension, and the prediction results have better fit and model performance.

Nondeterministic Features in Deep neural network design, training and inference

Neural networks are increasingly integral to scientific modeling and a wide range of real-world applications. However, standard neural networks often lack reliable certainty and confidence, and can be poorly calibrated, leading to nondeterministic results. To address these challenges, researchers have focused on understanding and quantifying uncertainty in neural network predictions. This work offers a comprehensive discussion of uncertainty estimation in neural networks, discussing recent advances, current challenges, and future research opportunities. This work explains key sources of uncertainty, challenges from model uncertainty and irreducible data uncertainty. We explore various methods for modeling these uncertainties, such as deterministic neural networks, Bayesian neural networks (BNNs), ensembles, and test-time data augmentation approaches. Additionally, the paper provides practical examples from fields like medical data analysis, robotics, and autonomous driving, illustrating the challenges and requirements associated with uncertainty in real-world applications. We also examine the limitations of current uncertainty quantification methods in safety-critical applications, and provide an outlook on future developments aimed at broader adoption of these methods in diverse domains. This paper serves as a valuable resource for both newcomers and those experienced in the field of uncertainty estimation in neural networks.

Predicting delamination in composite laminates through semi-analytical dynamic analysis and vibration-based quantitative assessment

Delamination is a frequent failure mode of laminated fiber-reinforced polymer (FRP) composites structure in aeronautical and other industries, leading to changes in vibration characteristics. Vibration-based techniques evaluate and compare the dynamic response between damaged and undamaged structures, and guarantee the non-destructive measurement with reliability and repeatability. Previous studies typically concentrate on using finite element method to obtain vibration characteristics and enhance the database for intelligent algorithms. This paper presents a semi-analytical result using the Chebyshev−Ritz method to expand delamination prediction. Vibration frequency serves as a global damage indicator, and multi-order frequency characteristics are utilized to identify the delamination length and location of FRP composite plates. A database of natural frequencies corresponding to damage parameters for FRP laminated plates is generated based on the established model using the region approach. An intelligent approach, known as a genetic algorithm optimization-based back-propagation (GA-BP) artificial neural network, is utilized for system identification. The network model is subjected to a sensitivity analysis, where artificial noise is added to vibration frequency to distinguish between the actual structure and the numerical model. The results indicate that the GA-BP algorithm shows good accuracy and stable performance against the standard neural networks for delamination analysis.

Ensuring Toplogical Data-Structure Preservation under Autoencoder Compression Due to Latent Space Regularization in Gauss–Legendre Nodes

We formulate a data-independent latent space regularization constraint for general unsupervised autoencoders. The regularization relies on sampling the autoencoder Jacobian at Legendre nodes, which are the centers of the Gauss–Legendre quadrature. Revisiting this classic allows us to prove that regularized autoencoders ensure a one-to-one re-embedding of the initial data manifold into its latent representation. Demonstrations show that previously proposed regularization strategies, such as contractive autoencoding, cause topological defects even in simple examples, as do convolutional-based (variational) autoencoders. In contrast, topological preservation is ensured by standard multilayer perceptron neural networks when regularized using our approach. This observation extends from the classic FashionMNIST dataset to (low-resolution) MRI brain scans, suggesting that reliable low-dimensional representations of complex high-dimensional datasets can be achieved using this regularization technique.

Intelligent risk identification for drilling lost circulation incidents using data-driven machine learning

In drilling operations, lost circulation (LC) poses a significant challenge, potentially leading to severe accidents such as overflow or blowouts if not accurately identified and controlled. This study introduces a method for identifying LC risk severity using data-driven machine learning and statistical algorithms, specifically demonstrated in the Ying-Qiong Basin of the South China Sea. The method begins with variance filtering to eliminate invalid data, followed by the application of the Pearson correlation coefficient to remove redundant data. This process results in the selection of 12 logging parameters out of the initial 38. Combined with indirect judgments of LC severity based on relevant prior knowledge, the data is labeled to construct the original dataset. A feature extraction and optimization method is proposed, which employs sliding time windows to extract 21 features. These features are then selected and optimized using a random forest algorithm to construct the optimal feature set. To enhance the controllability of the Extreme Learning Machine (ELM) model, the sparrow search algorithm is integrated. Additionally, several fitness functions are introduced for both regression and classification models, resulting in the development of the improved ELM (IELM) model. The dataset is split into training and test sets, and the performance of the IELM model is compared to that of the standard ELM and BP neural network models. The results indicate that the IELM model outperforms the other two, achieving an impressive test F1 score of 97.22 % for accurately identifying LC severity. Moreover, the model trained using the optimal feature set demonstrates higher accuracy and faster convergence. The proposed feature extraction and optimization method effectively increases the amount of information contained in the data, thereby enhancing the model's performance.

CGNSDE: Conditional Gaussian neural stochastic differential equation for modeling complex systems and data assimilation

A new knowledge-based and machine learning hybrid modeling approach, called conditional Gaussian neural stochastic differential equation (CGNSDE), is developed to facilitate modeling complex dynamical systems and implementing analytic formulae of the associated data assimilation (DA). In contrast to the standard neural network predictive models, the CGNSDE is designed to effectively tackle both forward prediction tasks and inverse state estimation problems. The CGNSDE starts by exploiting a systematic causal inference via information theory to build a simple knowledge-based nonlinear model that nevertheless captures as much explainable physics as possible. Then, neural networks are supplemented to the knowledge-based model in a specific way, which not only characterizes the remaining features that are challenging to model with simple forms but also advances the use of analytic formulae to efficiently compute the nonlinear DA solution. These analytic formulae are used as an additional computationally affordable loss to train the neural networks that directly improve the DA accuracy. This DA loss function promotes the CGNSDE to capture the interactions between state variables and thus advances its modeling skills. With the DA loss, the CGNSDE is more capable of estimating extreme events and quantifying the associated uncertainty. Furthermore, crucial physical properties in many complex systems, such as the translate-invariant local dependence of state variables, can significantly simplify the neural network structures and facilitate the CGNSDE to be applied to high-dimensional systems. Numerical experiments based on chaotic systems with intermittency and strong non-Gaussian features indicate that the CGNSDE outperforms knowledge-based regression models, and the DA loss further enhances the modeling skills of the CGNSDE.

Diurnal variation of indoor air pollutants and their influencing factors in educational buildings: A case study using LASSO-based ANNs

This study explores the diurnal variations and influencing factors of PM2.5, NO2, and ozone concentrations in educational buildings. Utilizing an integrated system of indoor and outdoor sensors, building automation control networks, and walk-through inspections, air quality data along with relevant characteristics were collected from ten educational buildings in Central Florida. Advanced Neural Network models (RNNs and CNNs), including the Long Short-Term Memory (LSTM) and the Attention Temporal Convolutional Network (ATCN) algorithms based on the Least Absolute Shrinkage and Selection Operator (LASSO), were developed to accurately identify diurnal patterns in indoor air quality (IAQ) and the differences in influencing factors. The findings indicate greater variability in diurnal differences and factors influencing indoor NO2 and ozone concentrations compared to PM2.5. Although the factors influencing day and night PM2.5 levels were similar, there were significant differences in the contribution weights of these factors. Optimized RNNs and CNNs significantly outperformed standard Artificial Neural Network (ANN) models in dynamically simulating and predicting target pollutants. Comparative analysis of the root-mean-square error (RMSE) demonstrated that LASSO-LSTM models comprehensively outperformed LASSO-ATCN models by averaging 13.4% (p < 0.05). These results can be referenced in studies concerning Indoor Air Quality (IAQ) control conducted in similar environmental settings.

Neural Networks for Geospatial Data

Analysis of geospatial data has traditionally been model-based, with a mean model, customarily specified as a linear regression on the covariates, and a Gaussian process covariance model, encoding the spatial dependence. While nonlinear machine learning algorithms like neural networks are increasingly being used for spatial analysis, current approaches depart from the model-based setup and cannot explicitly incorporate spatial covariance. We propose NN-GLS, embedding neural networks directly within the traditional Gaussian process (GP) geostatistical model to accommodate nonlinear mean functions while retaining all other advantages of GP, like explicit modeling of the spatial covariance and predicting at new locations via kriging. In NN-GLS, estimation of the neural network parameters for the nonlinear mean of the Gaussian Process explicitly accounts for the spatial covariance through use of the generalized least squares (GLS) loss, thus, extending the linear case. We show that NN-GLS admits a representation as a special type of graph neural network (GNN). This connection facilitates the use of standard neural network computational techniques for irregular geospatial data, enabling novel and scalable mini-batching, backpropagation, and kriging schemes. We provide methodology to obtain uncertainty bounds for estimation and predictions from NN-GLS. Theoretically, we show that NN-GLS will be consistent for irregularly observed spatially correlated data processes. We also provide a finite sample concentration rate, which quantifies the need to accurately model the spatial covariance in neural networks for dependent data. To our knowledge, these are the first large-sample results for any neural network algorithm for irregular spatial data. We demonstrate the methodology through numerous simulations and an application to air pollution modeling. We develop a software implementation of NN-GLS in the Python package geospaNN. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.

Composite Graph Neural Networks for Molecular Property Prediction.

Graph Neural Networks have proven to be very valuable models for the solution of a wide variety of problems on molecular graphs, as well as in many other research fields involving graph-structured data. Molecules are heterogeneous graphs composed of atoms of different species. Composite graph neural networks process heterogeneous graphs with multiple-state-updating networks, each one dedicated to a particular node type. This approach allows for the extraction of information from s graph more efficiently than standard graph neural networks that distinguish node types through a one-hot encoded type of vector. We carried out extensive experimentation on eight molecular graph datasets and on a large number of both classification and regression tasks. The results we obtained clearly show that composite graph neural networks are far more efficient in this setting than standard graph neural networks.

The attentive reconstruction of objects facilitates robust object recognition.

Humans are extremely robust in our ability to perceive and recognize objects-we see faces in tea stains and can recognize friends on dark streets. Yet, neurocomputational models of primate object recognition have focused on the initial feed-forward pass of processing through the ventral stream and less on the top-down feedback that likely underlies robust object perception and recognition. Aligned with the generative approach, we propose that the visual system actively facilitates recognition by reconstructing the object hypothesized to be in the image. Top-down attention then uses this reconstruction as a template to bias feedforward processing to align with the most plausible object hypothesis. Building on auto-encoder neural networks, our model makes detailed hypotheses about the appearance and location of the candidate objects in the image by reconstructing a complete object representation from potentially incomplete visual input due to noise and occlusion. The model then leverages the best object reconstruction, measured by reconstruction error, to direct the bottom-up process of selectively routing low-level features, a top-down biasing that captures a core function of attention. We evaluated our model using the MNIST-C (handwritten digits under corruptions) and ImageNet-C (real-world objects under corruptions) datasets. Not only did our model achieve superior performance on these challenging tasks designed to approximate real-world noise and occlusion viewing conditions, but also better accounted for human behavioral reaction times and error patterns than a standard feedforward Convolutional Neural Network. Our model suggests that a complete understanding of object perception and recognition requires integrating top-down and attention feedback, which we propose is an object reconstruction.

Event-Triggered Adaptive Neural Network Tracking Control for Uncertain Systems With Unknown Input Saturation Based on Command Filters.

This brief presents a modified event-triggered command filter backstepping tracking control scheme for a class of uncertain nonlinear systems with unknown input saturation based on the adaptive neural network (NN) technique. First, the virtual control functions are reconstructed to address the uncertainties in subsystems by using command filters. A piecewise continuous function is employed to deal with the unknown input saturation problem. Next, an event-triggered tracking controller is developed by utilizing the adaptive NN technique. Compared with standard NN control schemes based on multiple-function-approximators, our controller only requires a single NN. The closed-loop system stability is analyzed based on the Lyapunov stability theorem, and it is shown that the Zeno behavior is also avoided under the designed event-triggering mechanism. Simulation studies are performed to validate the effectiveness of our controller.

Weed species classification with UAV imagery and standard CNN models: Assessing the frontiers of training and inference phases

Accurate weed species identification is crucial for effective site-specific weed management (SSWM), enabling targeted and timely control measures for each weed in crop field. This study advanced the current approach to species-level weed identification during the early growth stage by integrating unmanned aerial vehicles (UAVs) imagery with standard convolutional neural networks (CNNs) models such as VGG16, Resnet152 and Inception-Resnet-v2. For this, a robust dataset was created with 33,467 labels of weeds (Atriplex patula, Chenopodium album, Convolvulus arvensis, Cyperus rotundus, Lolium rigidum, Portulaca oleracea, Salsola kali, Solanum nigrum) and crops (maize, tomato), which was subjected to different training, validation and test scenarios. Model inputs were adjusted in order to align them with the information represented by the UAV images. Initially, models were developed in balanced scenarios, gradually increasing label numbers to assess their performance. Inception-ResNet-v2 achieved over 90% accuracy with 400 labels, while ResNet152 and VGG16 required 600 and 800 labels, respectively, for similar accuracy. In a more complex and realistic scenarios with unbalanced datasets, Inception-ResNet-v2 outperformed, likely due to its deeper architecture and enhanced capability to capture intricate features and patterns within UAV images. The study emphasized the importance of the minority-to-majority species ratio in unbalanced datasets, which affects minority species classification. To prevent misclassification, it is crucial to determine the right number of labels for CNN model training and validation. Weed maps were generated after species classification using the Faster R–CNN algorithm as an object detector. This advancement in methodology facilitates the precise and efficient implementation of SSWM techniques.

Theory for Equivariant Quantum Neural Networks

Quantum neural network architectures that have little to no inductive biases are known to face trainability and generalization issues. Inspired by a similar problem, recent breakthroughs in machine learning address this challenge by creating models encoding the symmetries of the learning task. This is materialized through the usage of equivariant neural networks the action of which commutes with that of the symmetry. In this work, we import these ideas to the quantum realm by presenting a comprehensive theoretical framework to design equivariant quantum neural networks (EQNNs) for essentially any relevant symmetry group. We develop multiple methods to construct equivariant layers for EQNNs and analyze their advantages and drawbacks. Our methods can find unitary or general equivariant quantum channels efficiently even when the symmetry group is exponentially large or continuous. As a special implementation, we show how standard quantum convolutional neural networks (QCNNs) can be generalized to group-equivariant QCNNs where both the convolution and pooling layers are equivariant to the symmetry group. We then numerically demonstrate the effectiveness of a SU(2)-equivariant QCNN over symmetry-agnostic QCNN on a classification task of phases of matter in the bond-alternating Heisenberg model. Our framework can be readily applied to virtually all areas of quantum machine learning. Lastly, we discuss about how symmetry-informed models such as EQNNs provide hopes to alleviate central challenges such as barren plateaus, poor local minima, and sample complexity. Published by the American Physical Society 2024

SAF: Smart Aggregation Framework for Revealing Atoms Importance Rank and Improving Prediction Rates in Drug Discovery.

Machine learning, and representation learning in particular, has the potential to facilitate drug discovery by screening a large chemical space in silico. A successful approach for representing molecules is to treat them as graphs and utilize graph neural networks. One of the key limitations of such methods is the necessity to represent compounds with different numbers of atoms, which requires aggregating the atom's information. Common aggregation operators, such as averaging, result in a loss of information at the atom level. In this work, we propose a novel aggregating approach where each atom is weighted nonlinearly using the Boltzmann distribution with a hyperparameter analogous to temperature. We show that using this weighted aggregation improves the ability of the gold standard message-passing neural network to predict antibiotic activity. Moreover, by changing the temperature hyperparameter, our approach can reveal the atoms that are important for activity prediction in a smooth and consistent way, thus providing a novel regulated attention mechanism for graph neural networks. We further validate our method by showing that it recapitulates the functional group in β-lactam antibiotics. The ability of our approach to rank the atoms' importance for a desired function can be used within any graph neural network to provide interpretability of the results and predictions at the node level.

Prediction of dredged soil settlement based on improved BP neural network

Based on the measured settlement data from the Reclamation Project of Eastern Hengsha Shoal, two improved BP neural networks optimized by the particle swarm optimization (PSO) algorithm and genetic algorithm (GA) respectively are employed to predict the long-term consolidation settlement of the dredged soil in the reclamation area. By comparing the prediction results of the improved neural networks with those obtained from standard BP neural network and various curve fitting methods including hyperbola method, Hoshino method, and Asaoka method, as well as field-measured data, the improved neural network methods are validated to have a better predictive performance. The error analysis results indicate that compared with the standard BP neural network and curve-fitting prediction methods, the improved BP neural networks exhibit smaller prediction errors in terms of MAE, MSE, RMSE, MAPE, etc., and have a closer match between the predicted settlement values and the actual values. The selected GA-BP and PSO-BP neural network prediction methods in this study demonstrate higher prediction accuracy. Meanwhile, GA-BP demonstrates higher prediction accuracy and generalization ability than PSO-BP, but PSO-BP shows faster convergence in terms of prediction speed. The preferred methods presented in this paper exhibit good robustness and high reliability, which can achieve accurate prediction of dredged soil settlement. Additionally, unlike the traditional curve-fitting methods, the selected methods can effectively reduce the influence of subjective factors during the settlement prediction, and provide a new approach for settlement prediction, which can be effectively utilized as a substitute for long-term settlement monitoring and give reference for the design of settlement control of foundation of further construction projects.

Intelligent vehicle trajectory tracking control based on physics-informed neural network dynamics model

In order to solve the accuracy problem of trajectory tracking control method based on data-driven model, an intelligent vehicle trajectory tracking control method based on physics-informed neural network (PINN) vehicle dynamics model is proposed. Aiming at the problem of poor interpretability of data-driven model, a vehicle dynamics model based on the PINN is established, and the physics-driven deep learning method is used instead of the data-driven deep learning method to obtain the dynamic characteristics of the intelligent vehicle, to benefit from both the physical-based method and the data-driven method. A sequential training method is also proposed to solve the coupling problem when training multiple PINNs simultaneously. The model takes the nonlinearity of the neural network model and physical interpretability into consideration compared to the standard neural network model. Then, based on the PINN vehicle dynamics model, a trajectory tracking controller based on the iterative linear quadratic regulator (ILQR) control algorithm is developed. The optimal control law is derived by optimizing the ILQR control algorithm to implement the intelligent vehicle’s precise and stable tracking for the desired trajectory. The Levenberg-Marquardt (LM) algorithm and line search technology are used and damping factor adjustment rules are set up to enhance the convergence performance of the ILQR control algorithm. In order to verify the effectiveness of the proposed method, the simulation is conducted under the condition of double lane change. The simulation results demonstrate that the proposed method can track the reference trajectory accurately under the limited conditions. Its control performance is much better than other algorithms.

MTC-Net: Multi-scale feature fusion network for medical image segmentation

Image segmentation is critical in medical image processing for lesion detection, localisation, and subsequent diagnosis. Currently, computer-aided diagnosis (CAD) has played a significant role in improving diagnostic efficiency and accuracy. The segmentation task is made more difficult by the hazy lesion boundaries and uneven forms. Because standard convolutional neural networks (CNNs) are incapable of capturing global contextual information, adequate segmentation results are impossible to achieve. We propose a multiscale feature fusion network (MTC-Net) in this paper that integrates deep separable convolution and self-attentive modules in the encoder to achieve better local continuity of images and feature maps. In the decoder, a multi-branch multi-scale feature fusion module (MSFB) is utilized to improve the network’s feature extraction capability, and it is integrated with a global cooperative aggregation module (GCAM) to learn more contextual information and adaptively fuse multi-scale features. To develop rich hierarchical representations of irregular forms, the suggested detail enhancement module (DEM) adaptively integrates local characteristics with their global dependencies. To validate the effectiveness of the proposed network, we conducted extensive experiments, evaluated on the public datasets of skin, breast, thyroid and gastrointestinal tract with ISIC2018, BUSI, TN3K and Kvasir-SEG. The comparison with the latest methods also verifies the superiority of our proposed MTC-Net in terms of accuracy. Our code on https://github.com/gih23/MTC-Net.

Deep Learning and Multimodal Artificial Intelligence in Orthopaedic Surgery.

This review article focuses on the applications of deep learning with neural networks and multimodal neural networks in the orthopaedic domain. By providing practical examples of how artificial intelligence (AI) is being applied successfully in orthopaedic surgery, particularly in the realm of imaging data sets and the integration of clinical data, this study aims to provide orthopaedic surgeons with the necessary tools to not only evaluate existing literature but also to consider AI's potential in their own clinical or research pursuits. We first review standard deep neural networks which can analyze numerical clinical variables, then describe convolutional neural networks which can analyze image data, and then introduce multimodal AI models which analyze various types of different data. Then, we contrast these deep learning techniques with related but more limited techniques such as radiomics, describe how to interpret deep learning studies, and how to initiate such studies at your institution. Ultimately, by empowering orthopaedic surgeons with the knowledge and know-how of deep learning, this review aspires to facilitate the translation of research into clinical practice, thereby enhancing the efficacy and precision of real-world orthopaedic care for patients.