General Neural Network Research Articles

Extended ARMA graph neural networks for the prognosis of complex systems

With the advancement of intelligent sensing techniques, massive monitoring signals are collected and accumulated from industrial systems. Given that sensors are often correlated and constructed to reflect graph topology, the signals can be conceptualized as graph data. The polynomial filter-based graph neural networks (GNNs) are commonly employed to exploit information from nodes features and graph topology for graph data analysis. However, the polynomial filter-based GNNs encounter difficulty in accurately modeling sharp changes and the coefficients can vary a lot, making them hard to learn. To address this problem, a novel graph neural network named extended auto-regressive moving average graph neural network (eAGNN) is proposed. Compared with auto-regressive moving average (ARMA) neural network, the order restriction are removed, allowing for the inference of a more general neural network, which enables the modeling of filters with more different shapes. Furthermore, both low-frequency and high-frequency information are explicitly and separately extracted so as to alleviate the burden of the learning process and further enhance the learning capability. Finally, several experiments including public node classification and fault diagnosis were conducted. The results demonstrate that the proposed eAGNN exhibits high performance compared to alternative methods.

A general Physics-Informed Neural Network approach for deriving fluid flow fields from temperature distribution

One attractive application of Physics-Informed Neural Network (PINN) is deriving fluid flow information such as velocity and pressure from temperature field. However, this requires developing a suitable loss function for a given flow condition. In this paper we implemented a general loss function in PINN to directly obtain fluid velocity and pressure from temperature information. We first validated this method through cases of natural convection in a square cavity, flow around a cylinder in a square cavity, and two-dimensional channel flow. Additionally, we examined forced convection heat transfer around a cylinder in depth. Results show that different Reynolds numbers (2, 10, 40, and 140) can all be effectively resolved using temperature data. Furthermore, super-resolution predictions of temperature, velocity and pressure are achievable even far outside the training area, suggesting that the proposed PINN method could be used to measure fluid flow within confined spaces via finite visualization windows without tracers.

Hierarchical Stability Conditions for Two Types of Time-Varying Delay Generalized Neural Networks.

In this article, the stability analysis for generalized neural networks (GNNs) with a time-varying delay is investigated. About the delay, the differential has only an upper boundary or cannot be obtained. For the both two types of delayed GNNs, up to now, the second-order integral inequalities have been the highest-order integral inequalities utilized to derive the stability conditions. To establish the stability conditions on the basis of the high-order integral inequalities, two challenging issues are required to be resolved. One is the formulation of the Lyapunov-Krasovskii functional (LKF), the other is the high-degree polynomial negative conditions (NCs). By transforming the integrals in N-order generalized free-matrix-based integral inequalities (GFIIs) into the multiple integrals, the hierarchical LKFs are constructed by adopting these multiple integrals. Then, the novel modified matrix polynomial NCs are presented for the 2N-1 degree delay polynomials in the LKF differentials. Thus, the hierarchical linear matrix inequalities (LMIs) are set up and the nonlinear problems caused by the GFIIs are solved at the same time. Eventually, the superiority of the provided hierarchical stability criteria is demonstrated by several numeric examples.

GENet: A Generic Neural Network for Detecting Various Neurological Disorders From EEG

GENet: A Generic Neural Network for Detecting Various Neurological Disorders From EEG

Graph Neural Networks With High-Order Polynomial Spectral Filters.

General graph neural networks (GNNs) implement convolution operations on graphs based on polynomial spectral filters. Existing filters with high-order polynomial approximations can detect more structural information when reaching high-order neighborhoods but produce indistinguishable representations of nodes, which indicates their inefficiency of processing information in high-order neighborhoods, resulting in performance degradation. In this article, we theoretically identify the feasibility of avoiding this problem and attribute it to overfitting polynomial coefficients. To cope with it, the coefficients are restricted in two steps, dimensionality reduction of the coefficients' domain and sequential assignment of the forgetting factor. We transform the optimization of coefficients to the tuning of a hyperparameter and propose a flexible spectral-domain graph filter, which significantly reduces the memory demand and the adverse impacts on message transmission under large receptive fields. Utilizing our filter, the performance of GNNs is improved significantly in large receptive fields and the receptive fields of GNNs are multiplied as well. Meanwhile, the superiority of applying a high-order approximation is verified across various datasets, notably in strongly hyperbolic datasets. Codes are publicly available at: https://github.com/cengzeyuan/TNNLS-FFKSF.

DeepQCD: An end-to-end deep learning approach to quickest change detection

This paper aims to generalize the quickest change detection (QCD) framework via a data-driven approach. To this end, a generic neural network architecture is proposed for the QCD task, composed of feature transformation, recurrent, and dense layers. The neural network is trained end-to-end to learn the change detection rule directly from data without needing the knowledge of probabilistic data models. Specifically, the feature transformation layers can perform a broad range of operations including feature extraction, scaling, and normalization. The recurrent layers keep an internal state summarizing the time-series data seen so far and update the state as new information comes in. Finally, the dense layers map the internal state into a decision statistic, defined as the posterior probability that a change has taken place. Comparisons with the existing model-based QCD algorithms demonstrate the power of the proposed data-driven approach, called DeepQCD, under several scenarios including transient changes and temporally correlated data streams. Experiments with real-world data illustrate superior performance of DeepQCD compared to state-of-the-art algorithms in real-time anomaly detection over surveillance videos and real-time attack detection over Internet of Things (IoT) networks.

Projective Psychological Tests from the Point of View of Neural Network Theory of Society

It is shown that the issues raised by the international expert community in the discussion regarding the legality of using projective techniques (in particular, tests such as the Luscher test, the Szondi test and the Dellinger test) can be resolved on the basis of an adequate interpretation of the concept of “archetype” and an analogy with neural networks. At the same time, the interpretation of the concept “archetype” is based on the neural network theory of society, which allows us to remove the difficulties that arise with the traditional interpretation of this concept. It is shown that there is an equivalent scheme for the testing procedure, in which the testing itself can be reduced to the functioning of the external (output) layer of the general neural network, which also includes a neural network formed by the human brain. This approach makes it possible to remove many of the criticisms put forward against the use of projective techniques (even declaring them pseudoscientific), but it implies a significant change in existing approaches to the interpretation of results obtained when using tests of the type in question. Namely, within the framework of the proposed approach, not every possible combination of images on which the test is built makes sense. Such a meaning can be given only to some of them, specifically to those that are most often encountered in experiments. Such combinations, which can be interpreted as basic, are interpreted on the basis of an analogy with images recognized by a neural network. The remaining combinations are treated as images containing errors; they are reduced to the basic ones based on an analogy with the methods used in constructing error-resistant codes.

A Particle Swarm Optimization-Based Generative Adversarial Network

At present, the combination of general evolutionary algorithms (EAs) and neural networks is limited to optimizing the framework or hyper parameters of neural networks. To further extend applications of EAs on neural networks, we propose a particle swarm optimization (PSO) based generative adversarial network(GAN), named as PGAN in this paper. In the study, PSO is utilized as a generator to generate fake data, while the discriminator is a traditional fully connected neural network. In the confrontation process, when the proposed PSO can generate a better fake image, this will react to the discriminator, so that the discriminator can improve the recognition effect of the image and the better discriminator also accelerates the evolution of the overall model. Through experiments, we explore the new application value of EAs in deep learning, so that the sample data in EAs and the sample data in deep learning are interconnected. The PSO algorithm is improved, so that it truly participates in the confrontation with multi-layer perceptrons.

Potential Role of Generative Adversarial Networks in Enhancing Brain Tumors.

Contrast enhancement is necessary for visualizing, diagnosing, and treating brain tumors. Through this study, we aimed to examine the potential role of general adversarial neural networks in generating artificial intelligence-based enhancement of tumors using a lightweight model. A retrospective study was conducted on magnetic resonance imaging scans of patients diagnosed with brain tumors between 2020 and 2023. A generative adversarial neural network was built to generate images that would mimic the real contrast enhancement of these tumors. The performance of the neural network was evaluated quantitatively by VGG-16, ResNet, binary cross-entropy loss, mean absolute error, mean squared error, and structural similarity index measures. Regarding the qualitative evaluation, nine cases were randomly selected from the test set and were used to build a short satisfaction survey for experienced medical professionals. One hundred twenty-nine patients with 156 scans were identified from the hospital database. The data were randomly split into a training set and validation set (90%) and a test set (10%). The VGG loss function for training, validation, and test sets were 2,049.8, 2,632.6, and 4,276.9, respectively. Additionally, the structural similarity index measured 0.366, 0.356, and 0.3192, respectively. At the time of submitting the article, 23 medical professionals responded to the survey. The median overall satisfaction score was 7 of 10. Our network would open the door for using lightweight models in performing artificial contrast enhancement. Further research is necessary in this field to reach the point of clinical practicality.

An attention mechanism network based on the winner-take-all

In the field of neurology, neurons compete for brain attention in winner-take-all (WTA) competitions. Inspired by this, we propose a new WTA-based attention network (called ANW), which can be extended to general neural networks. The ANW network simulates WTA behavior in biological neurons; that is, the winner neuron itself becomes excited and sends its own spike signal. Furthermore, the winner will output an inhibition signal, which will be input into the cell synapses of other neurons. The ANW network regards the update of neuron weights after backpropagation as a stimulus input to neurons. Combined with the WTA function, the neural network will continue to update until it reaches a stable state. In this way, the ANW network can find the most active neurons in the network and generate the winners among these neurons to improve the ability of the neural network to extract features. As the ANW is a small and universal network, it can be seamlessly inserted into any neural network architecture, with negligible overhead, and can be trained together with a basic CNN. We verified the neural network with ANW by using the ImageNet-100, CIFAR-100 and MS COCO detection datasets for the experiments. The experimental results show that, compared with those of the baseline network, the top-1 err gains of 0.63% and 3.85% can be achieved for ANW, with classical ResNet-50 as the backbone network, respectively, demonstrating that the method can improve the performance of the model in image classification and target detection with wide applicability. The proposed method provides an effective, and almost consumption-neutral solution, for performance improvement in the field of computer vision target detection.

UVS-CNNs: Constructing general convolutional neural networks on quasi-uniform spherical images

Omnidirectional images, also known as spherical images, offer a significant advantage for the environmental sensing of mobile robots due to their wide field of view. However, previous studies of constructing convolutional neural networks on spherical images have been limited by non-uniform pixel sampling, leading to suboptimal performance in semantic segmentation. To address this issue, a novel pixel segmentation approach is proposed to achieve near-uniform pixel distribution across the entire spherical surface. The corresponding convolution operation for the resulting image is designed as well, which extends the capabilities of spherical CNNs from semantic segmentation to more complex tasks such as instance segmentation. The method is evaluated on the Stanford 2D3DS dataset and shows superior performance compared to conventional spherical CNNs. Furthermore, the method also achieves impressive instance segmentation results on our experimental LiDAR data, demonstrating the general feasibility of our approach for common CNN tasks. The related code and dataset are released in the following link: https://github.com/YoungRainy/UVS-U-Net.

Multiple feedback recurrent neural network based super-twisting predefined-time nonsingular terminal sliding mode control for quad-rotor UAV

This research deals with the overall predefined-time stability (PTS) of Unmanned Aerial Vehicles (UAV) ensuring rapid convergence based on a novel sliding mode control (SMC). The strength of predefined-time sliding manifolds lies in the convergence rate can be adjusted by an explicit parameter. For the limitation of chattering encountered by predefined-time SMC (PTSMC), a variable gain super-twisting algorithm (STA) with additional linear items is designed as the switch controller. To conserve the restrained computational resources of quadrotors, the equivalent control input is approximated by a multiple feedback recurrent neural network (MFRNN) directly, which is challenging for general recurrent neural networks. The proposed MFRNN is characterized by the incorporation of double-loop feedback within the layers, augmenting its capacity for accurate approximation. To address the vanishing gradients commonly encountered with traditional activation functions, LeakyRelu is chosen. The Lyapunov theory is utilized to ensure the PTS of overall system and obtain the MFRNN weight update laws. An experiment is conducted to validate the proposed scheme.

A New Hybrid Approach for Clustering, Classification, and Prediction of World Development Indicators Combining General Type-2 Fuzzy Systems and Neural Networks

A New Hybrid Approach for Clustering, Classification, and Prediction of World Development Indicators Combining General Type-2 Fuzzy Systems and Neural Networks

Deephullnet: a deep learning approach for solving the convex hull and concave hull problems with transformer

ABSTRACT Convex and concave hulls originating from computational geometry are widely applied in practice. For instance, to determine the boundaries of a geographical area within a group of cities, convex hulls can represent the approximate boundaries of the areas. Concave hulls can accurately describe the shape of the area. The traditional methods for solving two-dimensional convex hull problems include the Jarvis March algorithm and the Graham scanning algorithm. The K-nearest neighbours and alpha-shape methods are designed for solving the concave hull problem. Other than traditional algorithms, we consider using deep neural networks to handle the convex and concave hull problems. General neural networks cannot deal with the problem whose output is a variable-length sequence. Our study provides a machine-learning approach for solving the convex hull and concave hull problems. We combine the Pointer network (Ptr-Net) with the Transformer model and propose the DeepHullNet. The trained DeepHullNet can provide a feasible solution in the majority of cases. The experimental results show that DeepHullNet outperforms the original Ptr-Net regarding accuracy. Compared with traditional methods, DeepHullNet can generate a good solution quickly, and the running time is shortened by nearly 100 times based on GPU.

Machine learning for non-experts: A more accessible and simpler approach to automatic benthic habitat classification

Automating identification of benthic habitats from imagery, with Machine Learning (ML), is necessary to contribute efficiently and effectively to marine spatial planning. A promising method is to adapt pre-trained general convolutional neural networks (CNNs) to a new classification task (transfer learning). However, this is often inaccessible to a non-specialist, requiring large investments in computational resources and time (for user comprehension and model training). In this paper, we demonstrate a simpler transfer learning framework for classifying broad deep-sea benthic habitats. Specifically, we take an ‘off-the-shelf’ CNN (VGG16) and use it to extract features (pixel patterns) from benthic images (without further training). The default outputs of VGG16 are then fed in to a Support Vector Machine (SVM), a classical and simpler method than deep networks. For comparison, we also train the remaining classification layers of VGG16 using stochastic gradient descent. The discriminative power of these approaches is demonstrated on three benthic datasets (574–8353 images) from Norwegian waters; each using a unique imaging platform. Benthic habitats are broadly classified as Soft Substrate (sands, muds), Hard Substrate (gravels, cobbles and boulders) and Reef (Desmophyllum pertusum). We found that the relatively simplicity of the SVM classifier did not compromise performance. Results were competitive with the CNN classifier and consistently high, with test accuracy ranging from 0.87 to 0.95 (average = 0.9 (±0.04)) across datasets, somewhat increasing with dataset size. Impressively, these results were achieved 2.4–5× faster than CNN training and had significantly less dependency on high-specification hardware. Our suggested approach maximises conceptual and practical simplicity, representing a realistic baseline for novice users when approaching benthic habitat classification. This method has wide potential. It allows automated image grouping to aid annotation or further model selection, as well as screening of old-datasets. It is especially suited to offshore scenarios as it can provide quick, albeit crude, insights into habitat presence, allowing adaptation of sampling protocols in near real-time.

Asymptotic Stability Analysis and Stabilization Control for General Fractional-Order Neural Networks via an Unified Lyapunov Function

Asymptotic Stability Analysis and Stabilization Control for General Fractional-Order Neural Networks via an Unified Lyapunov Function

Prediction of pressure evolution in non-venting self-pressurized liquid hydrogen tanks using artificial neural network approach

Liquid hydrogen (LH2) is a future energy carrier compared to most conventional fuels. One of the practical bottlenecks in non-venting LH2 tanks is to accurately predict the pressure evolution. Accordingly, many thermal models and correlations were developed; however significant deviations arose due to inherent complex thermal phenomena which have not been addressed yet. A general artificial neural network (ANN) model has been first developed based on 448 datasets amassed from different literature sources to predict the pressure evolution under different heat fluxes, initial filling ratios, as well as tank volumes and shapes. The model has been configured with a typical multi-layer feed-forward structure using a back-propagation learning algorithm and Levenberg-Marquardt training algorithm. For the ANN model, a preliminary optimization process has been conducted for reaching the optimal values for training-to-testing datasets ratio, learning parameters and optimum network structure. Besides, some criteria were set to evaluate the ANN model performance. The ANN model with the configuration of 5-11-11-1 was found to be adopted for such a prediction task. Compared to experimental results and literature models and correlations, the ANN model has been proven to predict the pressure evolution in non-venting self-pressurized LH2 tanks with a mean square error of 1.1779 × 10−5 s2, root mean square error of 0.0034 s, mean absolute error of 6%, mean relative error of 0.2% as well as a satisfactory correlation coefficient of 0.99934 between the network outputs and corresponding targets. This research work provides an effective modeling approach based on artificial neural network to solve one complex problem related with LH2 tanks with fast, accurate and consistent results. Also, the results of the present work can offer some practical guidance for the design of LH2 tanks. Further, the present work gives some insights into new application areas of ANN which will be the emphasis of future research work.

Automatic tooth arrangement with joint features of point and mesh representations via diffusion probabilistic models

Tooth arrangement is a crucial step in orthodontics treatment, in which aligning teeth could improve overall well-being, enhance facial aesthetics, and boost self-confidence. To improve the efficiency of tooth arrangement and minimize errors associated with unreasonable designs by inexperienced practitioners, some deep learning-based tooth arrangement methods have been proposed. Currently, most existing approaches employ MLPs to model the nonlinear relationship between tooth features and transformation matrices to achieve tooth arrangement automatically. However, the limited datasets (which to our knowledge, have not been made public) collected from clinical practice constrain the applicability of existing methods, making them inadequate for addressing diverse malocclusion issues. To address this challenge, we propose a general tooth arrangement neural network based on the diffusion probabilistic model. Conditioned on the features extracted from the dental model, the diffusion probabilistic model can learn the distribution of teeth transformation matrices from malocclusion to normal occlusion by gradually denoising from a random variable, thus more adeptly managing real orthodontic data. To take full advantage of effective features, we exploit both mesh and point cloud representations by designing different encoding networks to extract the tooth (local) and jaw (global) features, respectively. In addition to traditional metrics ADD, PA-ADD, CSA, and MErot, we propose a new evaluation metric based on dental arch curves to judge whether the generated teeth meet the individual normal occlusion. Experimental results demonstrate that our proposed method achieves state-of-the-art tooth alignment results and satisfactory occlusal relationships between dental arches. We will publish the code and dataset.

Reactor field reconstruction from sparse and movable sensors using Voronoi tessellation-assisted convolutional neural networks

The aging of operational reactors leads to increased mechanical vibrations in the reactor interior. The vibration of the in-core sensors near their nominal locations is a new problem for neutronic field reconstruction. Current field-reconstruction methods fail to handle spatially moving sensors. In this study, we propose a Voronoi tessellation technique in combination with convolutional neural networks to handle this challenge. Observations from movable in-core sensors were projected onto the same global field structure using Voronoi tessellation, holding the magnitude and location information of the sensors. General convolutional neural networks were used to learn maps from observations to the global field. The proposed method reconstructed multi-physics fields (including fast flux, thermal flux, and power rate) using observations from a single field (such as thermal flux). Numerical tests based on the IAEA benchmark demonstrated the potential of the proposed method in practical engineering applications, particularly within an amplitude of 5 cm around the nominal locations, which led to average relative errors below 5% and 10% in the L2\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$L_2$$\\end{document} and L∞\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$L_{\\infty }$$\\end{document} norms, respectively.

Neural Network Approximation for Pessimistic Offline Reinforcement Learning

Deep reinforcement learning (RL) has shown remarkable success in specific offline decision-making scenarios, yet its theoretical guarantees are still under development. Existing works on offline RL theory primarily emphasize a few trivial settings, such as linear MDP or general function approximation with strong assumptions and independent data, which lack guidance for practical use. The coupling of deep learning and Bellman residuals makes this problem challenging, in addition to the difficulty of data dependence. In this paper, we establish a non-asymptotic estimation error of pessimistic offline RL using general neural network approximation with C-mixing data regarding the structure of networks, the dimension of datasets, and the concentrability of data coverage, under mild assumptions. Our result shows that the estimation error consists of two parts: the first converges to zero at a desired rate on the sample size with partially controllable concentrability, and the second becomes negligible if the residual constraint is tight. This result demonstrates the explicit efficiency of deep adversarial offline RL frameworks. We utilize the empirical process tool for C-mixing sequences and the neural network approximation theory for the Holder class to achieve this. We also develop methods to bound the Bellman estimation error caused by function approximation with empirical Bellman constraint perturbations. Additionally, we present a result that lessens the curse of dimensionality using data with low intrinsic dimensionality and function classes with low complexity. Our estimation provides valuable insights into the development of deep offline RL and guidance for algorithm model design.