Neural Network Structure Research Articles

Generating Descriptive Text From Images - Image Caption Generator

In the modern era, image captioning has become one of the most widely required tools. Moreover, there are inbuilt applications that generate and provide a caption for a certain image, all these things are done with the help of deep neural network models. The process of generating a description of an image is called image captioning. It requires recognizing the important objects, their attributes, and the relationships among the objects in an image. It generates syntactically and semantically correct sentences. In this paper, we present a deep learning model to describe images and generate captions using computer vision and machine translation. This paper aims to detect different objects found in an image, recognize the relationships between those objects and generate captions. The dataset used is Flickr8k and the programming language used was Python3, and an ML technique called Transfer Learning will be implemented with the help of the caption model, to demonstrate the proposed experiment. This paper will also elaborate on the functions and structure of the various Neural networks involved. Generating image captions is an important aspect of Computer Vision and Natural language processing. Image caption generators can find applications in Image segmentation as used by Facebook and Google Photos, and even more so, its use can be extended to video frames. They will easily automate the job of a person who has to interpret images. Not to mention it has immense scope in helping visually impaired people

Research on Isomorphic Task Transfer Algorithm Based on Knowledge Distillation in Multi-Agent Collaborative Systems.

In response to the increasing number of agents and changing task scenarios in multi-agent collaborative systems, existing collaborative strategies struggle to effectively adapt to new task scenarios. To address this challenge, this paper proposes a knowledge distillation method combined with a domain separation network (DSN-KD). This method leverages the well-performing policy network from a source task as the teacher model, utilizes a domain-separated neural network structure to correct the teacher model's outputs as supervision, and guides the learning of agents in new tasks. The proposed method does not require the pre-design or training of complex state-action mappings, thereby reducing the cost of transfer. Experimental results in scenarios such as UAV surveillance and UAV cooperative target occupation, robot cooperative box pushing, UAV cooperative target strike, and multi-agent cooperative resource recovery in a particle simulation environment demonstrate that the DSN-KD transfer method effectively enhances the learning speed of new task policies and improves the proximity of the policy model to the theoretically optimal policy in practical tasks.

Predictive Analytics for Demand Forecasting: A deep Learning-based Decision Support System

Demand forecasting is a critical component of supply chain management and business operations, enabling organizations to make informed decisions about production, inventory management, and resource allocation. In recent years, predictive analytics has emerged as a powerful tool for enhancing the accuracy and efficiency of demand forecasting. This review paper explores the transformative role of predictive analytics and deep learning in demand forecasting. It examines how these advanced techniques have evolved from traditional models based on past sales data, offering nuanced predictions through sophisticated statistical and machine learning methods. Deep learning, with its neural network structures, brings automatic feature learning, complex pattern handling, and scalability, enhancing forecasting in sectors like retail, manufacturing, and healthcare. The paper reviews various deep learning models, compares them with traditional methods, and discusses their impact on business operations and decision-making. It concludes by looking at future trends in predictive analytics and deep learning in demand forecasting.

An Intelligent Financial Fraud Detection Model Using Knowledge Graph-Integrated Deep Neural Network

Financial fraud detection has been an urgent technical demand in cyberspace. It highly relies on clear extraction and deep representation toward complex relationships inside financial social networks. As consequence, this study combines both knowledge graph and deep learning to deal with such issue. Thus, an intelligent financial fraud detection model based on knowledge graph guidance and deep neural network is proposed in this paper. First, a new knowledge graph based on financially related systems is constructed, which includes multiple entities and relationships. Then, an adversarial learning-based neural network structure is formulated to extract financial attributes. Finally, the detection results can be output according to the extracted factors. Empirically, the proposal is implemented on a real-world dataset for performance evaluation. The experimental results show that it has higher accuracy and effectiveness compared to traditional fraud detection methods. The proposed detection model can not only identify known fraudulent behaviors, but also predict potential fraud patterns based on consumer habits, thereby improving the security and reliability of financial transactions. It can also update the knowledge graph in real-time, enabling it to cope with emerging fraud methods and variants.

CvT-UNet: A weld pool segmentation method integrating a CNN and a transformer

In the acquisition of weld seam images, the presence of light and impurity interference often leads to weak edge problems in weld seam images, making the identification of weld seam defects challenging. In such scenarios, segmentation processing must be performed specifically at the fusion pool of the weld seam. In recent years, Transformers, which leverage self-attention mechanisms, have demonstrated superior sequence modeling capabilities compared to convolutional neural networks (CNNs). The Transformer excels at weighting different positions within a sequence, enabling effective handling of long-range dependencies. However, Transformers lack the ability of CNNs to extract local information. To address this limitation, this paper introduces a novel CvT-UNet semantic segmentation model that combines the strengths of both the convolutional neural network and the transformer. The model combines the global context information capabilities of the Transformer with the spatial information advantages provided by the convolutional neural network. The proposed model employs an encoder-decoder architecture by building a U-shaped network with designed CvT blocks. Depthwise separable convolutions are utilized to balance parameter quantities, and the skip-connection module is redesigned to enable the model to achieve more precise segmentation of weld seam fusion pools with fewer parameters. Experimental evaluations on diverse weld seam datasets under different environmental conditions demonstrate promising results. The model achieves average intersection over union (IoU) values of 93.75 %, 88.31 %, and 90.86 %, respectively. Specifically, on an automotive weld seam dataset, CvT-UNet outperforms the purely convolutional neural network structure UNet3+, with a 1.63 % increase in the mean IoU (MIOU) and a 1.39 % increase in the mean pixel accuracy (MPA). Compared to the TransUNet fusion model, CvT-UNet results in a 0.33 % increase in the MIOU and a 0.6 % increase in the MPA. Furthermore, compared to the state-of-the-art curve segmentation model LIOT, CvT-UNet achieves a 1.94 % increase in the MIOU and a 0.61 % increase in the MPA, demonstrating favorable segmentation performance.

Intelligent inversion of mesoscale eddy temperature anomaly profiles based on multi-source remote sensing data

Inversion of the underwater temperature anomaly of mesoscale eddies from sea surface remote sensing data plays an important role in understanding the three-dimensional structure of eddies. In this article, a neural network structure, named the eddy temperature anomaly inversion network (EddyTAINet), is proposed and established to invert the vertical temperature anomalies of eddies using 20-year (2002–2022) Argo profiles and multi-source remote sensing data in the northwestern Pacific Ocean. Vertical temperature anomaly profiles with 45 depth levels ranging from 10 to 1000 m are derived as the outputs of the inversion networks. The input feature sequences are extracted from the mesoscale eddy product based on sea level anomaly (SLA) data, and sea surface temperature (SST) data are also introduced into the proposed network structure to improve its accuracy. For the problem of inconsistent sizes of the 2D matrix SST data corresponding to mesoscale eddies, the spatial pyramid pooling (SPP) network module is introduced, which allows different sizes of SST data to be simply convolved and then subjected to hierarchical feature extraction to obtain fixed-size feature vectors. Validation and comparison of the network structure are performed and show the improved performance of EddyTAINet. The average root mean square error (RMSE) and mean absolute error (MAE) are 0.63 °C/0.66 °C and 0.48 °C/0.51 °C for cyclonic and anticyclonic eddies, respectively, which are more than 20% lower than the results of the network using the SLA-based data.

A Genetic Algorithm and Fuzzy Neural Network-Based Intelligent Temperature Control Decision Model for Coagulation Cooling Systems

The coagulation cooling system is a common key component in many industrial processes, and reasonable temperature control is crucial. However, due to the complexity of the coagulation cooling system, traditional temperature control methods often cannot achieve optimal performance. To solve this problem, we design an intelligent temperature control decision model by a combination of genetic algorithm and fuzzy neural network. The study firstly utilizes genetic algorithm to optimize the objective function and constraint conditions of the coagulation cooling PID system. At the same time, fuzzy neural network is fused with genetic algorithm to establish a dedicated T-S/2 neural network structure, completing the complete model design of this study. Finally, cooling efficiency, task completion rate and model stability analysis are evaluated on real-world datasets. To validate the proposed model, an example of a coagulation cooling system was constructed in the laboratory and compared with traditional temperature control methods. The experimental results show that the proposal can significantly improve the performance of temperature control and reduce energy consumption under different conditions. In addition, the proposal has the characteristics of adaptability and optimization performance, and can effectively achieve optimal temperature control in uncertain and complex environments.

A deep learning approach for solving the stationary compositional two-phase equilibrium problems

In this paper, we propose and investigate a deep neural network approach for solving the stationary compositional two-phase equilibrium problems in porous media. A recent approach is the unified formulation advocated by Lauser et al. (2011) which contains the complementarity conditions. The advantage of this formulation lies in its potential to handle the appearance and disappearance of phases automatically. To solve numerically the system of equations, a new strategy called NPIPM (NonParametric Interior-Point Method) is proposed by Vu et al. (2021). However, the method still has some disadvantages preventing the convergence to a solution. Taking inspiration from the work of Raissi et al. (2019) for the study of nonlinear partial differential equations using Physics-Informed Neural Networks (PINN), we realize that we can apply a deep neural network to obtain the solution of the stationary compositional model containing gas and liquid. We design a deep neural network structure and define a new loss function for the problem. The effectiveness of the proposed method is validated through numerical results obtained from a series of systems with increasing components. This machine learning approach provides a promising idea to solve the curse of dimensionality issue in the sense of the increasing number of components in stationary problems.

Power-enhanced residual network for function approximation and physics-informed inverse problems

In this study, we investigate how the updating of weights during forward operation and the computation of gradients during backpropagation impact the optimization process, training procedure, and overall performance of the neural network, particularly the multi-layer perceptrons (MLPs). This paper introduces a novel neural network structure called the Power-Enhancing residual network, inspired by highway network and residual network, designed to improve the network's capabilities for both smooth and non-smooth functions approximation in 2D and 3D settings. By incorporating power terms into residual elements, the architecture enhances the stability of weight updating, thereby facilitating better convergence and accuracy. The study explores network depth, width, and optimization methods, showing the architecture's adaptability and performance advantages. Consistently, the results emphasize the exceptional accuracy of the proposed Power-Enhancing residual network, particularly for non-smooth functions. Real-world examples also confirm its superiority over plain neural network in terms of accuracy, convergence, and efficiency. Moreover, the proposed architecture is also applied to solving the inverse Burgers' equation, demonstrating superior performance. In conclusion, the Power-Enhancing residual network offers a versatile solution that significantly enhances neural network capabilities by emphasizing the importance of stable weight updates for effective training in deep neural networks. The codes implemented are available at: https://github.com/CMMAi/ResNet_for_PINN.

Improving deep neural network random initialization through neuronal rewiring

The deep learning literature is continuously updated with new architectures and training techniques. However, weight initialization is overlooked by most recent research, despite some intriguing findings regarding random weights. On the other hand, recent works have been approaching Network Science to understand the structure and dynamics of Artificial Neural Networks (ANNs) after training. Therefore, in this work, we analyze the centrality of neurons in randomly initialized networks. We show that a higher neuronal strength variance may decrease performance, while a lower neuronal strength variance usually improves it. A new method is then proposed to rewire neuronal connections according to a preferential attachment (PA) rule based on their strength, which significantly reduces the strength variance of layers initialized by common methods. In this sense, PA rewiring only reorganizes connections, while preserving the magnitude and distribution of the weights. We show through an extensive statistical analysis on image classification tasks that performance is improved in most cases, both during training and testing, when using both simple and complex architectures and learning schedules. Our results show that, aside from the magnitude, the organization of the weights is also relevant for better initialization of deep ANNs.

Car-following informed neural networks for real-time vehicle trajectory imputation and prediction

Vehicle trajectory information is a crucial part of improving the efficiency and the safety of the ITS. Data missing or irregular sampling in the real-world road traffic makes it hard to obtain accurate real-time vehicle trajectories. In this paper, we focus on trajectory imputation and prediction tasks with small data (magnitude set as 10 1 and 10 2 ). Limited by insufficient data, the simulation results of the existing data-driven algorithms are unsatisfactory. With car-following models integrated as prior physical information to constrain the training process, we design the car-following-informed neural network (CFINN). A multi-head self-attention layer is attached to the fully connected network layer to extract vehicle features. Different from the structure of most neural networks in regression analysis, an extra physics-based dataset is constructed in the CFINN. The loss function consists of two parts including the given trajectory's fitting error and the generated trajectory's residual error. We embed the gated recurrent unit-based encoder–decoder layer to the CFINN framework for trajectory predictions. The rationality and the superiority of our model are validated on the NGSIM dataset and the HighD dataset. Compared with baseline models in both single-vehicle and queue-typed trajectory imputation experiments, lower error can be achieved via the CFINN and coefficients of car-following models can be calibrated. According to driving regimes derived from CFINN-based trajectory prediction experiments, we discuss the impact of cut-in behaviours on the target vehicle and carry out the kinetic analysis. The novel neural network model driven by both data and physical knowledge provides technical support in vehicle status assessments and trajectory predictions.

Greedy deep stochastic configuration networks ensemble with boosting negative correlation learning

Deep stochastic configuration networks (DSCNs) employ data-dependent supervision mechanism to randomly assign node parameters and incrementally construct the deep neural network structure, thereby ensuring the model's universal approximation property. To build a random neural networks ensemble model with better generalization performance, we propose a novel greedy deep stochastic configuration networks ensemble model based on boosting negative correlation learning, termed as GDSCNE. Firstly, greedy optimization strategy based on inequality constraints is utilized to generate random parameters of base components with multi-layer architecture, which can accelerate the decline of network residuals when configuring a new node. Additionally, boosting negative correlation learning framework is presented for the base components ensemble process, which uses least square approach with negative correlation learning penalty term to update the ensemble output weights for each base component, subsequently, boosting method is applied to construct a stronger ensemble model by adaptive weighting through the results of base components. Finally, we evaluated GDSCNE on the popular regression benchmark datasets from the KEEL, experimental results demonstrate that GDSCNE outperforms state-of-the-art random learning algorithms in terms of regression accuracy and generalization performance across several regression datasets with varying sizes.

CGNSDE: Conditional Gaussian neural stochastic differential equation for modeling complex systems and data assimilation

A new knowledge-based and machine learning hybrid modeling approach, called conditional Gaussian neural stochastic differential equation (CGNSDE), is developed to facilitate modeling complex dynamical systems and implementing analytic formulae of the associated data assimilation (DA). In contrast to the standard neural network predictive models, the CGNSDE is designed to effectively tackle both forward prediction tasks and inverse state estimation problems. The CGNSDE starts by exploiting a systematic causal inference via information theory to build a simple knowledge-based nonlinear model that nevertheless captures as much explainable physics as possible. Then, neural networks are supplemented to the knowledge-based model in a specific way, which not only characterizes the remaining features that are challenging to model with simple forms but also advances the use of analytic formulae to efficiently compute the nonlinear DA solution. These analytic formulae are used as an additional computationally affordable loss to train the neural networks that directly improve the DA accuracy. This DA loss function promotes the CGNSDE to capture the interactions between state variables and thus advances its modeling skills. With the DA loss, the CGNSDE is more capable of estimating extreme events and quantifying the associated uncertainty. Furthermore, crucial physical properties in many complex systems, such as the translate-invariant local dependence of state variables, can significantly simplify the neural network structures and facilitate the CGNSDE to be applied to high-dimensional systems. Numerical experiments based on chaotic systems with intermittency and strong non-Gaussian features indicate that the CGNSDE outperforms knowledge-based regression models, and the DA loss further enhances the modeling skills of the CGNSDE.

An explainable neural network integrating Jiles-Atherton and nonlinear auto-regressive exogenous models for modeling universal hysteresis

The inherent nonlinear and memory-dependent input-output characteristics of piezoelectric actuators pose challenges to the precision of piezoelectric positioning systems. In order to solve this problem, this paper firstly transforms the Jiles-Atherton (JA) model into a neural network structure, designs the Jiles-Atherton neural network (JANN), and combines JANN with nonlinear autoregressive exogenous input (NARX) neural network. A hybrid JA-NARX neural network model is proposed for the first time. This model has the advantages of simple structure, high modeling accuracy, and good interpretability. The effectiveness of the proposed JA-NARX neural network model is validated through a series of experiments, specifically assessing its capacity to accurately capture rate-dependent and asymmetric hysteresis characteristics. The results show that although the proposed neural network model has fewer layers and relatively simple structure, it can realize the high-precision modeling of piezoelectric hysteresis dynamics at a lower computational cost. The experimental data shows that, under the excitation of 60 Hz input signal, the model's PV error only accounts for 0.82% of the full scale range, and the modeling performance is far superior to other models.

Data-Driven Smart Assessment for Enterprise Audit Risks Based on Radial Base Function Neural Network and Grey Correlation Analysis

In the era of big data, audit risk assessment is facing increasing business volume. Hence, it is important to develop automatic audit risk assessment methods for enterprises with the assistance of intelligent algorithms. As a consequence, this paper proposes data-driven smart assessment for enterprise audit risks-based radial basis function (RBF) neural networks and grey correlation analysis (GCA). First, evaluation data are collected and analyzed to understand the characteristics and influencing factors of enterprise audit risks. Secondly, RBF interpolation is introduced to establish the network structure for RBF neural networks. Then, GCA is integrated with RBF Interpolation (RBFNN) to formulate the automatic audit risk evaluation method, and the proposal is named RBF and GCA. Such a combined methodology can improve audit risk assessment efficiency. Finally, we make some experiments to evaluate the proposed TBF-GCA on a real-world dataset, and some evaluation indicators are specified based on the actual audit risk situation of the enterprises. The obtained results reveal that RBF–GCA has high accuracy in identity precision and can effectively identify audit risks for enterprises.

Data-driven methods for the inverse problem of suspension system excited by jump and diffusion stochastic track excitation

Following a lengthy tenure of service, suspension systems may exhibit inaccurate model parameters due to the presence of complex conditions, which can result in a deterioration of control and the potential for operational failures. It is therefore imperative to estimate the key parameters of suspension systems. This paper introduces two data-driven methods for addressing the inverse problem in suspension systems subjected to stochastic track excitations. By employing physics-informed neural networks (PINNs) and Monte Carlo (MC) simulation, we are able to address the resulting integro-differential equation that arises from stochastic jump processes, thus avoiding the necessity for mesh grids. In order to mitigate the numerical challenges that arise from the system parameters, a residual-based adaptive sampling method is proposed. These methods effectively infer unknown parameters, addressing scenarios where data is directly available as a probability density function (PDF) or only sparse trajectories are given. In the latter case, a novel loss function employing Kullback-Leibler divergence facilitates learning from stochastic trajectories. Both methods successfully obtain solutions to the forward Kolmogorov equation, as validated by numerical experiments testing their robustness against added noise. The results demonstrate accurate parameter estimation under varying noise intensities, highlighting the methods’ robustness.

Application of artificial neural networks for indirect load measurements on bridges from vibration measurements

The German Research Foundation (DFG) has launched in September 2022 the priority program SPP 2388 100+ to develop new methods for digital representation, SHM and lifetime management of complex structures, due to the continuously increasing amount of old infrastructure buildings. The present contribution is prepared within the LEMOTRA project as a part of SPP 2388 100+. The data assimilation techniques present a potential SHM approach and presume a permanent update of a computational model by use of continuous measurements. System changes resulting from damage or aging processes can be detected and localized, provided the measurement and model prediction share the same cause. Thus, the load identification is a necessary prerequisite for data assimilation methods. In this paper an approach using artificial neural networks is developed to determine the main parameters of the traffic load associated with vehicles crossing a bridge. Simple feed forward neural networks have already been used for predicting structural displacements or discovering system anomalies. However, a more sophisticated network structure is necessary for the purpose of extracting features from time history measurement data. Therefore, a cluster structure of convolutional neural networks (CNN) was developed to gain knowledge about load characteristics such as load magnitudes, load velocities or the number of vehicles on the bridge. The developed approach has been tested on a numerical example. For this purpose, a finite element model of a bridge is created and loaded by various moving loads with variable relevant parameters. Simulated acceleration data at multiple locations on the bridge are considered as artificial measurements and then used in the training process. The impact of different measurement noise levels on the quality of the CNNs is being investigated as well. The obtained results allow for some conclusions on advantages and drawbacks which should be a matter of discussion during the conference.

Network Traffic Classification Against Adversarial Attacks Based Deep Learning

Background: Cybersecurity is a prominent concern in today's interconnected world, encompassing both local and remote wireless and wired access across diverse communication technology platforms. It is important to recognize the threat posed by hackers who are currently compromising organizational functionalities, bypassing security measures, and stealing hypersensitive information. Common hacking techniques such as Portscan, Distributed Denial of Service (DDoS) attacks, and the use of Botnets, are frequently utilized by hackers. Materials and Methods: The authors explore the advantages of using Machine Learning (ML) to classify attacks such as Port Scanning, DDoS, Botnet, and Botnet-Attempt in a mixture of both benign and attack traffic. They use various Artificial Neural Networks (ANN) structures, each having distinct properties, to train and test on a benchmark dataset (CICIDS2017). The aim is to identify the most effective ANN model and the optimal number of input features required to classify data that contains events of Portscan, DDoS, Botnet, and Botnet-Attempt attacks. Results: Various features are used as inputs for an ML model with single and multiple hidden layers, each with different neurons, to evaluate their impact on classification accuracy using a Python language. The best accuracy obtained is 99.71%, achieved by using all features of the dataset and 4 hidden layers, while the accuracy obtained using only 7 features is 97.6%. Conclusion: ANN models can perform well in classifying network traffic against adversarial attacks by using an optimal combination of features as input and hidden layers.

A Robust Deep Learning Framework Based on Spectrograms for Heart Sound Classification.

Heart sound analysis plays an important role in early detecting heart disease. However, manual detection requires doctors with extensive clinical experience, which increases uncertainty for the task, especially in medically underdeveloped areas. This paper proposes a robust neural network structure with an improved attention module for automatic classification of heart sound wave. In the preprocessing stage, noise removal with Butterworth bandpass filter is first adopted, and then heart sound recordings are converted into time-frequency spectrum by short-time Fourier transform (STFT). The model is driven by STFT spectrum. It automatically extracts features through four down sample blocks with different filters. Subsequently, an improved attention module based on Squeeze-and-Excitation module and coordinate attention module is developed for feature fusion. Finally, the neural network will give a category for heart sound waves based on the learned features. The global average pooling layer is adopted for reducing the model's weight and avoiding overfitting, while focal loss is further introduced as the loss function to minimize the data imbalance problem. Validation experiments have been conducted on two publicly available datasets, and the results well demonstrate the effectiveness and advantages of our method.

KerbNet: A QoE-Aware Kernel-Based Backdoor Attack Framework

Deep neural networks are vulnerable to backdoor attacks, where a specially-designed trigger will lead to misclassification of any benign samples. However, existing backdoor attacks usually impose conspicuous patch triggers on images, which are easily detected by humans and defense algorithms. Existing works on invisible triggers, however, either have reduced attack success rate or yield detectable patterns to visual inspections. In this paper, we propose <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">KerbNet</small> , a kernel-based backdoor attack framework, which applies kernel operations to clean samples as the trigger to incur misclassification. The kernel-processed samples achieve a high attack success rate while appearing natural with high Quality-of-Experience (QoE). We carefully design the kernel trigger generation algorithm by exploiting the neural network structure to propagate the influence of the trigger to the target misclassification label under the QoE constraint. We conduct extensive experiments on five datasets, i.e., MNIST, GTSRB, CIFAR-10, CelebA, and ImageNette to evaluate the effectiveness and practicality of <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">KerbNet</small> under the impact of various factors, including neuron-residing layer, kernel size, base image, loss function, model structure, and so on. We also show that our proposed attacks can evade state-of-the-art defense strategies and visual inspections. Code will be available after publication.