Standard Neural Network Research Articles

Towards the characterization of representations learned via capsule-based network architectures

Capsule Neural Networks (CapsNets) have been re-introduced as a more compact and interpretable alternative to standard deep neural networks. While recent efforts have proved their compression capabilities, to date, their interpretability properties have not been fully assessed. Here, we conduct a systematic and principled study towards assessing the interpretability of these types of networks. We pay special attention towards analyzing the level to which part-whole relationships are encoded within the learned representation. Our analysis in the MNIST, SVHN, CIFAR-10, and CelebA datasets on several capsule-based architectures suggest that the representations encoded in CapsNets might not be as disentangled nor strictly related to parts-whole relationships as is commonly stated in the literature.

Ultra-wide low-frequency bandgap characteristics of auxeticity-based composite resonator for elastic wave manipulation and machine learning-based inverse structural design

Although auxetic metamaterials are generally light in weight, strong in compression and superior in energy absorption, less research has focused on their low-frequency bandgap realization and inverse structural design. Moreover, broadening the width of bandgap of auxetic metamaterials keep challenging as well in practical low-frequency applications. To broaden their applications, this work provides a new auxeticity-based composite resonator design by filling hard engineering material into the soft auxetic structure perforated by orthogonally-arrayed peanut-shaped holes to achieve ultra-wide low-frequency bandgap. The soft material is silicone rubber and the hard material may be concrete, steel or plumbum. Firstly, the wave propagation mechanics of the present composite structure is studied and then the bandgap characteristics are thoroughly investigated by parameter analysis. On this basis, a machine learning (ML) method combining the standard back-propagation neural network and genetic algorithm is proposed for bandgap prediction and inverse structural design. Results indicate that the present composite structure performs well in generating ultra-wide low-frequency bandgap, i.e. [11.35, 24.07] Hz, and the relative bandgap width reaches 71.76 %. Moreover, the present ML method is effective in the forward prediction of bandgap and the inverse structure design fitting the specific bandgap requirement. For the created optimal designs, the ML method and the finite element simulation can give similar results, and the maximum relative error between them is just about 5 %, which verifies the accuracy of the ML design method. This work provides a new way to realize the structural design of auxeticity-based acoustic metamaterials with low-frequency and broad bandgap.

A Comparative Study of Swin Transformer and CNN Models for Crowd Counting

Abstract. Crowd counting, a critical component in the management and safety planning of large gatherings and public spaces, is essential for ensuring smooth event operations and preventing potential overcrowding issues. While the standard convolutional neural network (CNN) based model performs well in head counting tasks, it has certain drawbacks when applied to complex scenarios. With the rapid development of artificial intelligence, Transformer models that rely on self-attention mechanisms, as Swin Transformer, have demonstrated exceptional performance in visual tasks, such as image classification, and segmentation in recent times. This study examines the experimental findings of Swin Transformer's head counting tasks and contrasts them with the CNN-based model. Mean Absolute Error (MAE) and Mean Square Error (MSE) evaluation indicators show that the Transformer model outperforms the classic CNN model in terms of generalization abilities when dealing with complicated scenarios. Future research work will increase the diversity of data sets and focus on optimizing model structure and improving training efficiency.

Use of ResNets for HLB Disease Detection on Orange Leaves Using Terrestrial Multispectral Images

Abstract. Huanglongbing (HLB) is a bacterial disease transmitted by different vectors of sap-sucking insects. It affects all crops of citrus trees, decreasing the values of those fruits in the market and eventually the decay of orchards. In Brazil, the world's leading orange producer, citriculture faces severe issues with HLB and substantial economic loss. Technical means of scanning the orchards with high-throughput becomes essential for the sustainability of this industry. In this study, we propose to investigate an operational strategy consisting of scanning large portions of foliage (the canopy of one tree or more) in which there can few early foliage symptoms. It is proposed to investigate deep learning tools to solve this complex binary classification problem. The study is based on a dataset comprising 1,297 terrestrial multispectral (14 channels) images captured at high spatial resolution in a commercial orange orchard in Brazil. It is proposed to adapt and retrain standard neural network architectures, namely ResNets18 and ResNets34, to process such images. Our analysis reveals promising results, with both models demonstrating convergence and achieving stable performance. Notably, ResNet18 outperformed ResNet34, achieving an accuracy of 76.45% compared to 66.79% from ResNet34. These findings suggest that deep neural network methods can effectively manage non-radiometrically calibrated data and accurately distinguish images with HLB symptoms from healthy plants . However, with reduced datasets and limited possibilities for transfer learning and fine-tuning, it seems that only reasonable sized networks can be trained. Thus, more advanced state-of-the-art tools of the are still challenging to deploy for agricultural multi- or hyperspectral data.

A Stacked Neural Network Model for Damage Localization.

Traditional vibration-based damage detection methods often involve human intervention in decision-making, therefore being time-consuming and error-prone. In this study, we propose using Artificial Neural Networks (ANNs) to detect patterns in the structural response and create accurate predictions. The features extracted from the response signal are the Relative Frequency Shifts (RFSs) of the first eight weak-axis bending vibration modes, and the predictions refer to the damage location. To increase the accuracy of the predictions, we propose a novel stacked neural network approach, capable of detecting damage locations with high accuracy. The dataset used for training involves, as input data, the RFSs calculated with an original method for numerous damage locations and severities. The following models were used as building blocks for our stacked approach: Multilayer Perceptron, Recurrent Neural Network, Long Short-term Memory, and Gated Recurrent Units. The entire beam was thus split into segments and each network was trained in this stacked model on one beam segment. All results obtained with the models are also compared to a standard neural network trained on the entire beam. The results obtained show that the model that performs the best contains 14 stacked two-layer feedforward networks.

Tree Sequences as a General-Purpose Tool for Population Genetic Inference.

As population genetic data increase in size, new methods have been developed to store genetic information in efficient ways, such as tree sequences. These data structures are computationally and storage efficient but are not interchangeable with existing data structures used for many population genetic inference methodologies such as the use of convolutional neural networks applied to population genetic alignments. To better utilize these new data structures, we propose and implement a graph convolutional network to directly learn from tree sequence topology and node data, allowing for the use of neural network applications without an intermediate step of converting tree sequences to population genetic alignment format. We then compare our approach to standard convolutional neural network approaches on a set of previously defined benchmarking tasks including recombination rate estimation, positive selection detection, introgression detection, and demographic model parameter inference. We show that tree sequences can be directly learned from using a graph convolutional network approach and can be used to perform well on these common population genetic inference tasks with accuracies roughly matching or even exceeding that of a convolutional neural network-based method. As tree sequences become more widely used in population genetic research, we foresee developments and optimizations of this work to provide a foundation for population genetic inference moving forward.

Distance preserving machine learning for uncertainty aware accelerator capacitance predictions

Abstract Accurate uncertainty estimations are essential for producing reliable machine learning models, especially in safety-critical applications such as accelerator systems. Gaussian process models are generally regarded as the gold standard for this task; however, they can struggle with large, high-dimensional datasets. Combining deep neural networks with Gaussian process approximation techniques has shown promising results, but dimensionality reduction through standard deep neural network layers is not guaranteed to maintain the distance information necessary for Gaussian process models. We build on previous work by comparing the use of the singular value decomposition against a spectral-normalized dense layer as a feature extractor for a deep neural Gaussian process approximation model and apply it to a capacitance prediction problem for the High Voltage Converter Modulators in the Oak Ridge Spallation Neutron Source. Our model shows improved distance preservation and predicts in-distribution capacitance values with less than 1% error.

Fractional gradient optimized explainable convolutional neural network for Alzheimer's disease diagnosis

Alzheimer's is one of the brain syndromes that steadily affects the brain memory. The early stage of Alzheimer's disease (AD) is referred to as mild cognitive impairment (MCI), and the growth of Alzheimer's is not certain in patients with MCI. The premature detection of Alzheimer's is crucial for maintaining healthy brain function and avoiding memory loss. Different multi-neural network architectures have been proposed by researchers for efficient and accurate AD detection. The absence of improved feature extraction mechanisms and unexplored efficient optimizers in complex benchmark architectures lead to an inefficient and inaccurate AD classification. Moreover, the standard convolutional neural network (CNN)-based architectures for Alzheimer's diagnosis lack interpretability in their predictions. An interpretable, simplified, yet effective deep learning model is required for the accurate classification of AD. In this study, a generalized fractional order-based CNN classifier with explainable artificial intelligence (XAI) capabilities is proposed for accurate, efficient, and interpretable classification of AD diagnosis. The proposed study (a) classifies AD accurately by incorporating unexplored pooling technique with enhanced feature extraction mechanism, (b) provides fractional order-based optimization approach for adaptive learning and fast convergence speed, and (c) suggests an interpretable method for proving the transparency of the model. The proposed model outperforms complex benchmark architectures with regard to accuracy using standard ADNI dataset. The proposed fractional order-based CNN classifier achieves an improved accuracy of 99 % as compared to the state-of-the-art models.

Prediction method for 4D trajectory of large fixed wing UAV under multi-layer convolution and bidirectional gating model

Abstract With the increase of low-altitude flight tasks, the importance of research on low-altitude flight safety and efficient operation has gradually become prominent. Therefore, it is necessary to try to establish a low-altitude airspace flight path pre-planning method based on 4D trajectory prediction, which can effectively support the above research. In order to improve the integrity, accuracy and reliability of the long-range trajectory prediction results of large fixed wing UAV (LFW UAV), based on the standard neural network as the basic concept to form a fusion model of improved convolutional neural network (CNN) and bidirectional gated recurrent neural network (BIGRU). At the same time, the flight trajectory fitting preprocessing is added to build the completed multilayer cyclic convolutional improvement model (MCCI). The experimental results show that the MCCI model has obvious advantages in terms of prediction accuracy, deviation range and predictable duration when dealing with the LFW UAV 4D trajectory prediction problem, and better reflects the characteristics of 3D position and spatiotemporal elements. The comparative analysis shows that the error values of the MCCI model are smaller than those of the comparison prediction models in any dimension, and the prediction results have better fit and model performance.

Nondeterministic Features in Deep neural network design, training and inference

Neural networks are increasingly integral to scientific modeling and a wide range of real-world applications. However, standard neural networks often lack reliable certainty and confidence, and can be poorly calibrated, leading to nondeterministic results. To address these challenges, researchers have focused on understanding and quantifying uncertainty in neural network predictions. This work offers a comprehensive discussion of uncertainty estimation in neural networks, discussing recent advances, current challenges, and future research opportunities. This work explains key sources of uncertainty, challenges from model uncertainty and irreducible data uncertainty. We explore various methods for modeling these uncertainties, such as deterministic neural networks, Bayesian neural networks (BNNs), ensembles, and test-time data augmentation approaches. Additionally, the paper provides practical examples from fields like medical data analysis, robotics, and autonomous driving, illustrating the challenges and requirements associated with uncertainty in real-world applications. We also examine the limitations of current uncertainty quantification methods in safety-critical applications, and provide an outlook on future developments aimed at broader adoption of these methods in diverse domains. This paper serves as a valuable resource for both newcomers and those experienced in the field of uncertainty estimation in neural networks.

Predicting delamination in composite laminates through semi-analytical dynamic analysis and vibration-based quantitative assessment

Delamination is a frequent failure mode of laminated fiber-reinforced polymer (FRP) composites structure in aeronautical and other industries, leading to changes in vibration characteristics. Vibration-based techniques evaluate and compare the dynamic response between damaged and undamaged structures, and guarantee the non-destructive measurement with reliability and repeatability. Previous studies typically concentrate on using finite element method to obtain vibration characteristics and enhance the database for intelligent algorithms. This paper presents a semi-analytical result using the Chebyshev−Ritz method to expand delamination prediction. Vibration frequency serves as a global damage indicator, and multi-order frequency characteristics are utilized to identify the delamination length and location of FRP composite plates. A database of natural frequencies corresponding to damage parameters for FRP laminated plates is generated based on the established model using the region approach. An intelligent approach, known as a genetic algorithm optimization-based back-propagation (GA-BP) artificial neural network, is utilized for system identification. The network model is subjected to a sensitivity analysis, where artificial noise is added to vibration frequency to distinguish between the actual structure and the numerical model. The results indicate that the GA-BP algorithm shows good accuracy and stable performance against the standard neural networks for delamination analysis.

Ensuring Toplogical Data-Structure Preservation under Autoencoder Compression Due to Latent Space Regularization in Gauss–Legendre Nodes

We formulate a data-independent latent space regularization constraint for general unsupervised autoencoders. The regularization relies on sampling the autoencoder Jacobian at Legendre nodes, which are the centers of the Gauss–Legendre quadrature. Revisiting this classic allows us to prove that regularized autoencoders ensure a one-to-one re-embedding of the initial data manifold into its latent representation. Demonstrations show that previously proposed regularization strategies, such as contractive autoencoding, cause topological defects even in simple examples, as do convolutional-based (variational) autoencoders. In contrast, topological preservation is ensured by standard multilayer perceptron neural networks when regularized using our approach. This observation extends from the classic FashionMNIST dataset to (low-resolution) MRI brain scans, suggesting that reliable low-dimensional representations of complex high-dimensional datasets can be achieved using this regularization technique.

Intelligent risk identification for drilling lost circulation incidents using data-driven machine learning

In drilling operations, lost circulation (LC) poses a significant challenge, potentially leading to severe accidents such as overflow or blowouts if not accurately identified and controlled. This study introduces a method for identifying LC risk severity using data-driven machine learning and statistical algorithms, specifically demonstrated in the Ying-Qiong Basin of the South China Sea. The method begins with variance filtering to eliminate invalid data, followed by the application of the Pearson correlation coefficient to remove redundant data. This process results in the selection of 12 logging parameters out of the initial 38. Combined with indirect judgments of LC severity based on relevant prior knowledge, the data is labeled to construct the original dataset. A feature extraction and optimization method is proposed, which employs sliding time windows to extract 21 features. These features are then selected and optimized using a random forest algorithm to construct the optimal feature set. To enhance the controllability of the Extreme Learning Machine (ELM) model, the sparrow search algorithm is integrated. Additionally, several fitness functions are introduced for both regression and classification models, resulting in the development of the improved ELM (IELM) model. The dataset is split into training and test sets, and the performance of the IELM model is compared to that of the standard ELM and BP neural network models. The results indicate that the IELM model outperforms the other two, achieving an impressive test F1 score of 97.22 % for accurately identifying LC severity. Moreover, the model trained using the optimal feature set demonstrates higher accuracy and faster convergence. The proposed feature extraction and optimization method effectively increases the amount of information contained in the data, thereby enhancing the model's performance.

CGNSDE: Conditional Gaussian neural stochastic differential equation for modeling complex systems and data assimilation

A new knowledge-based and machine learning hybrid modeling approach, called conditional Gaussian neural stochastic differential equation (CGNSDE), is developed to facilitate modeling complex dynamical systems and implementing analytic formulae of the associated data assimilation (DA). In contrast to the standard neural network predictive models, the CGNSDE is designed to effectively tackle both forward prediction tasks and inverse state estimation problems. The CGNSDE starts by exploiting a systematic causal inference via information theory to build a simple knowledge-based nonlinear model that nevertheless captures as much explainable physics as possible. Then, neural networks are supplemented to the knowledge-based model in a specific way, which not only characterizes the remaining features that are challenging to model with simple forms but also advances the use of analytic formulae to efficiently compute the nonlinear DA solution. These analytic formulae are used as an additional computationally affordable loss to train the neural networks that directly improve the DA accuracy. This DA loss function promotes the CGNSDE to capture the interactions between state variables and thus advances its modeling skills. With the DA loss, the CGNSDE is more capable of estimating extreme events and quantifying the associated uncertainty. Furthermore, crucial physical properties in many complex systems, such as the translate-invariant local dependence of state variables, can significantly simplify the neural network structures and facilitate the CGNSDE to be applied to high-dimensional systems. Numerical experiments based on chaotic systems with intermittency and strong non-Gaussian features indicate that the CGNSDE outperforms knowledge-based regression models, and the DA loss further enhances the modeling skills of the CGNSDE.

Diurnal variation of indoor air pollutants and their influencing factors in educational buildings: A case study using LASSO-based ANNs

This study explores the diurnal variations and influencing factors of PM2.5, NO2, and ozone concentrations in educational buildings. Utilizing an integrated system of indoor and outdoor sensors, building automation control networks, and walk-through inspections, air quality data along with relevant characteristics were collected from ten educational buildings in Central Florida. Advanced Neural Network models (RNNs and CNNs), including the Long Short-Term Memory (LSTM) and the Attention Temporal Convolutional Network (ATCN) algorithms based on the Least Absolute Shrinkage and Selection Operator (LASSO), were developed to accurately identify diurnal patterns in indoor air quality (IAQ) and the differences in influencing factors. The findings indicate greater variability in diurnal differences and factors influencing indoor NO2 and ozone concentrations compared to PM2.5. Although the factors influencing day and night PM2.5 levels were similar, there were significant differences in the contribution weights of these factors. Optimized RNNs and CNNs significantly outperformed standard Artificial Neural Network (ANN) models in dynamically simulating and predicting target pollutants. Comparative analysis of the root-mean-square error (RMSE) demonstrated that LASSO-LSTM models comprehensively outperformed LASSO-ATCN models by averaging 13.4% (p < 0.05). These results can be referenced in studies concerning Indoor Air Quality (IAQ) control conducted in similar environmental settings.

Neural Networks for Geospatial Data

Analysis of geospatial data has traditionally been model-based, with a mean model, customarily specified as a linear regression on the covariates, and a Gaussian process covariance model, encoding the spatial dependence. While nonlinear machine learning algorithms like neural networks are increasingly being used for spatial analysis, current approaches depart from the model-based setup and cannot explicitly incorporate spatial covariance. We propose NN-GLS, embedding neural networks directly within the traditional Gaussian process (GP) geostatistical model to accommodate nonlinear mean functions while retaining all other advantages of GP, like explicit modeling of the spatial covariance and predicting at new locations via kriging. In NN-GLS, estimation of the neural network parameters for the nonlinear mean of the Gaussian Process explicitly accounts for the spatial covariance through use of the generalized least squares (GLS) loss, thus, extending the linear case. We show that NN-GLS admits a representation as a special type of graph neural network (GNN). This connection facilitates the use of standard neural network computational techniques for irregular geospatial data, enabling novel and scalable mini-batching, backpropagation, and kriging schemes. We provide methodology to obtain uncertainty bounds for estimation and predictions from NN-GLS. Theoretically, we show that NN-GLS will be consistent for irregularly observed spatially correlated data processes. We also provide a finite sample concentration rate, which quantifies the need to accurately model the spatial covariance in neural networks for dependent data. To our knowledge, these are the first large-sample results for any neural network algorithm for irregular spatial data. We demonstrate the methodology through numerous simulations and an application to air pollution modeling. We develop a software implementation of NN-GLS in the Python package geospaNN. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.

Composite Graph Neural Networks for Molecular Property Prediction.

Graph Neural Networks have proven to be very valuable models for the solution of a wide variety of problems on molecular graphs, as well as in many other research fields involving graph-structured data. Molecules are heterogeneous graphs composed of atoms of different species. Composite graph neural networks process heterogeneous graphs with multiple-state-updating networks, each one dedicated to a particular node type. This approach allows for the extraction of information from s graph more efficiently than standard graph neural networks that distinguish node types through a one-hot encoded type of vector. We carried out extensive experimentation on eight molecular graph datasets and on a large number of both classification and regression tasks. The results we obtained clearly show that composite graph neural networks are far more efficient in this setting than standard graph neural networks.

The attentive reconstruction of objects facilitates robust object recognition.

Humans are extremely robust in our ability to perceive and recognize objects-we see faces in tea stains and can recognize friends on dark streets. Yet, neurocomputational models of primate object recognition have focused on the initial feed-forward pass of processing through the ventral stream and less on the top-down feedback that likely underlies robust object perception and recognition. Aligned with the generative approach, we propose that the visual system actively facilitates recognition by reconstructing the object hypothesized to be in the image. Top-down attention then uses this reconstruction as a template to bias feedforward processing to align with the most plausible object hypothesis. Building on auto-encoder neural networks, our model makes detailed hypotheses about the appearance and location of the candidate objects in the image by reconstructing a complete object representation from potentially incomplete visual input due to noise and occlusion. The model then leverages the best object reconstruction, measured by reconstruction error, to direct the bottom-up process of selectively routing low-level features, a top-down biasing that captures a core function of attention. We evaluated our model using the MNIST-C (handwritten digits under corruptions) and ImageNet-C (real-world objects under corruptions) datasets. Not only did our model achieve superior performance on these challenging tasks designed to approximate real-world noise and occlusion viewing conditions, but also better accounted for human behavioral reaction times and error patterns than a standard feedforward Convolutional Neural Network. Our model suggests that a complete understanding of object perception and recognition requires integrating top-down and attention feedback, which we propose is an object reconstruction.

Event-Triggered Adaptive Neural Network Tracking Control for Uncertain Systems With Unknown Input Saturation Based on Command Filters.

This brief presents a modified event-triggered command filter backstepping tracking control scheme for a class of uncertain nonlinear systems with unknown input saturation based on the adaptive neural network (NN) technique. First, the virtual control functions are reconstructed to address the uncertainties in subsystems by using command filters. A piecewise continuous function is employed to deal with the unknown input saturation problem. Next, an event-triggered tracking controller is developed by utilizing the adaptive NN technique. Compared with standard NN control schemes based on multiple-function-approximators, our controller only requires a single NN. The closed-loop system stability is analyzed based on the Lyapunov stability theorem, and it is shown that the Zeno behavior is also avoided under the designed event-triggering mechanism. Simulation studies are performed to validate the effectiveness of our controller.

Weed species classification with UAV imagery and standard CNN models: Assessing the frontiers of training and inference phases

Accurate weed species identification is crucial for effective site-specific weed management (SSWM), enabling targeted and timely control measures for each weed in crop field. This study advanced the current approach to species-level weed identification during the early growth stage by integrating unmanned aerial vehicles (UAVs) imagery with standard convolutional neural networks (CNNs) models such as VGG16, Resnet152 and Inception-Resnet-v2. For this, a robust dataset was created with 33,467 labels of weeds (Atriplex patula, Chenopodium album, Convolvulus arvensis, Cyperus rotundus, Lolium rigidum, Portulaca oleracea, Salsola kali, Solanum nigrum) and crops (maize, tomato), which was subjected to different training, validation and test scenarios. Model inputs were adjusted in order to align them with the information represented by the UAV images. Initially, models were developed in balanced scenarios, gradually increasing label numbers to assess their performance. Inception-ResNet-v2 achieved over 90% accuracy with 400 labels, while ResNet152 and VGG16 required 600 and 800 labels, respectively, for similar accuracy. In a more complex and realistic scenarios with unbalanced datasets, Inception-ResNet-v2 outperformed, likely due to its deeper architecture and enhanced capability to capture intricate features and patterns within UAV images. The study emphasized the importance of the minority-to-majority species ratio in unbalanced datasets, which affects minority species classification. To prevent misclassification, it is crucial to determine the right number of labels for CNN model training and validation. Weed maps were generated after species classification using the Faster R–CNN algorithm as an object detector. This advancement in methodology facilitates the precise and efficient implementation of SSWM techniques.