Convergence Of Neural Network Research Articles

Recurrent Neural Network (RNN) Inference Cloud Computing

The purpose of this technical article is to investigate the convergence of Recurrent Neural Networks (RNNs) with Cloud Computing considering the potential benefits and challenges that may be encountered in bringing these two systems together. Recurrent Neural Networks (RNNs) constitute a particular type of artificial neural network that is capable of processing data sequentially in such a way that it becomes well-suited for applications that involve handling time series or language processing. On the other hand, cloud computing provides computer infrastructures on a scalable basis as well as flexible computation services that are accessible whenever they are needed. This paper targets making RNN applications more efficient and faster with the use of RNN libraries on cloud computing platforms through which the cloud services can be used for things like data processing power and storage. By combining RNNs with Cloud Computing, this paper aims to enhance the efficiency and performance of RNN-based applications by leveraging the cloud's computational power and storage capabilities. The study investigates the impact of deploying RNN inference tasks in the cloud environment, analyzing factors such as latency, cost-effectiveness, and scalability.

Gradient-based adaptive neural network technique for two-dimensional local fractional elliptic PDEs

Abstract This paper introduces gradient-based adaptive neural networks to solve local fractional elliptic partial differential equations. The impact of physics-informed neural networks helps to approximate elliptic partial differential equations governed by the physical process. The proposed technique employs learning the behaviour of complex systems based on input-output data, and automatic differentiation ensures accurate computation of gradient. The method computes the singularity-embedded local fractional partial derivative model on a Hausdorff metric, which otherwise halts the computation by available approximating numerical methods. This is possible because the new network is capable of updating the weight associated with loss terms depending on the solution domain and requirement of solution behaviour. The semi-positive definite character of the neural tangent kernel achieves the convergence of gradient-based adaptive neural networks. The importance of hyperparameters, namely the number of neurons and the learning rate, is shown by considering a stationary anomalous diffusion-convection model on a rectangular domain. The proposed method showcases the network’s ability to approximate solutions of various local fractional elliptic partial differential equations with varying fractal parameters.

Investigating the Evolving Landscape of Deepfake Technology: Generative AI's Role in it's Generation and Detection

The world of artificial intelligence is constantly changing, with Generative AI and Large Language Models (LLMs) leading the way in bringing new technological advancements. This paper offers a detailed look at these groundbreaking technologies and how they are shaping the digital world today. We explore the technical aspects of Generative AI and LLMs, explain their unique features, and compare them to traditional AI models.One of the key focuses of our research is the growing issue of DeepFakes—artificial intelligence-generated media that presents a significant challenge in verifying content. We conduct a thorough examination of few deepfake detection techniques out of which we will be implementing and analyzing one of them. Our research implements a framework for Deep Fake Image Detection. The suggested solution utilizes a RESNET-50(Residual Network with 50 layers) and MTCNN (Multi-task Cascaded Convolutional Networks) models for detecting whether the images are real or fake. This study conducts the Hypothesis testing for the proposed solution taking in consideration that the current Deepfake detection algorithms are less effective in detecting highly realistic Deepfakes compared to less sophisticated manipulations. By investigating the convergence of deep learning, neural networks, and sophisticated algorithms, we set the stage for advancements in AI-based content verification.

Can neural networks benefit from objectives that encourage iterative convergent computations? A case study of ResNets and object classification.

Recent work has suggested that feedforward residual neural networks (ResNets) approximate iterative recurrent computations. Iterative computations are useful in many domains, so they might provide good solutions for neural networks to learn. However, principled methods for measuring and manipulating iterative convergence in neural networks remain lacking. Here we address this gap by 1) quantifying the degree to which ResNets learn iterative solutions and 2) introducing a regularization approach that encourages the learning of iterative solutions. Iterative methods are characterized by two properties: iteration and convergence. To quantify these properties, we define three indices of iterative convergence. Consistent with previous work, we show that, even though ResNets can express iterative solutions, they do not learn them when trained conventionally on computer-vision tasks. We then introduce regularizations to encourage iterative convergent computation and test whether this provides a useful inductive bias. To make the networks more iterative, we manipulate the degree of weight sharing across layers using soft gradient coupling. This new method provides a form of recurrence regularization and can interpolate smoothly between an ordinary ResNet and a "recurrent" ResNet (i.e., one that uses identical weights across layers and thus could be physically implemented with a recurrent network computing the successive stages iteratively across time). To make the networks more convergent we impose a Lipschitz constraint on the residual functions using spectral normalization. The three indices of iterative convergence reveal that the gradient coupling and the Lipschitz constraint succeed at making the networks iterative and convergent, respectively. To showcase the practicality of our approach, we study how iterative convergence impacts generalization on standard visual recognition tasks (MNIST, CIFAR-10, CIFAR-100) or challenging recognition tasks with partial occlusions (Digitclutter). We find that iterative convergent computation, in these tasks, does not provide a useful inductive bias for ResNets. Importantly, our approach may be useful for investigating other network architectures and tasks as well and we hope that our study provides a useful starting point for investigating the broader question of whether iterative convergence can help neural networks in their generalization.

A modified U-Net convolutional neural network for segmenting periprostatic adipose tissue based on contour feature learning

ObjectiveThis study trains a U-shaped fully convolutional neural network (U-Net) model based on peripheral contour measures to achieve rapid, accurate, automated identification and segmentation of periprostatic adipose tissue (PPAT). MethodsCurrently, no studies are using deep learning methods to discriminate and segment periprostatic adipose tissue. This paper proposes a novel and modified, U-shaped convolutional neural network contour control points on a small number of datasets of MRI T2W images of PPAT combined with its gradient images as a feature learning method to reduce feature ambiguity caused by the differences in PPAT contours of different patients. This paper adopts a supervised learning method on the labeled dataset, combining the probability and spatial distribution of control points, and proposes a weighted loss function to optimize the neural network's convergence speed and detection performance. Based on high-precision detection of control points, this paper uses a convex curve fitting to obtain the final PPAT contour. The imaging segmentation results were compared with those of a fully convolutional network (FCN), U-Net, and semantic segmentation convolutional network (SegNet) on three evaluation metrics: Dice similarity coefficient (DSC), Hausdorff distance (HD), and intersection over union ratio (IoU). ResultsCropped images with a 270 × 270-pixel matrix had DSC, HD, and IoU values of 70.1%, 27 mm, and 56.1%, respectively; downscaled images with a 256 × 256-pixel matrix had 68.7%, 26.7 mm, and 54.1%. A U-Net network based on peripheral contour characteristics predicted the complete periprostatic adipose tissue contours on T2W images at different levels. FCN, U-Net, and SegNet could not completely predict them. ConclusionThis U-Net convolutional neural network based on peripheral contour features can identify and segment periprostatic adipose tissue quite well. Cropped images with a 270 × 270-pixel matrix are more appropriate for use with the U-Net convolutional neural network based on contour features; reducing the resolution of the original image will lower the accuracy of the U-Net convolutional neural network. FCN and SegNet are not appropriate for identifying PPAT on T2 sequence MR images. Our method can automatically segment PPAT rapidly and accurately, laying a foundation for PPAT image analysis.

Empowering robust biometric authentication: The fusion of deep learning and security image analysis

Many societal entities now have more excellent standards for the efficacy and dependability of identification systems due to the ongoing advancement of computer technology. Traditional identification methods, such as keys and smart cards, have been supplanted by biometric systems in highly secure environments. This research presents a smart computational method for automatically authenticating fingerprints for identity (ID) verification and personal identification. Compared to more traditional machine learning algorithms, the results from applying Deep learning (DL) in areas like computer vision, image identification, robotics, and voice processing have generally been positive. Due to their capacity to analyse big data size and deal with fluctuations in biometric data (such as ageing or expression problems), DL has been heavily used by the artificial intelligence research community. Several biometric systems have succeeded with automatic feature extraction employing deep learning approaches like Convolutional Neural Networks (CNNs). In this research, we provide a biometric process that uses convolutional neural networks. This work introduces a deep learning-based biometric identification system that uses Monte Carlo Dropout (MC Dropout). Combining these two systems makes the authentication process more secure and dependable. Fingerprint image enhancement techniques involve the application of Gabor filters and structure-adaptive anisotropic filters, which have proven to be effective in enhancing the clarity and distinctiveness of fingerprint patterns. To improve the efficiency of deep learning models, this work proposes the Inception-Augmentation GAN (IAGAN) model for data augmentation. The study adds to security development by integrating novel biometric identification and authentication approaches with cutting-edge neural network technology. In this research, we provide a new activation function to speed up the convergence of deep neural networks. The results of 99.6% on Gabor filters and 99.8% on the structure-adaptive anisotropic filter with GACNN with MCD show that deep neural networks can excel over competing approaches with enough training data.

Optimization Control of Adaptive Traffic Signal with Deep Reinforcement Learning

The optimization and control of traffic signals is very important for logistics transportation. It not only improves the operational efficiency and safety of road traffic, but also conforms to the direction of the intelligent, green, and sustainable development of modern cities. In order to improve the optimization effect of traffic signal control, this paper proposes a traffic signal optimization method based on deep reinforcement learning and Simulation of Urban Mobility (SUMO) software for urban traffic scenarios. The intersection training scenario was established using SUMO micro traffic simulation software, and the maximum vehicle queue length and vehicle queue time were selected as performance evaluation indicators. In order to be more relevant to the real environment, the experiment uses Weibull distribution to simulate vehicle generation. Since deep reinforcement learning takes into account both perceptual and decision-making capabilities, this study proposes a traffic signal optimization control model based on the deep reinforcement learning Deep Q Network (DQN) algorithm by considering the realism and complexity of traffic intersections, and first uses the DQN algorithm to train the model in a training scenario. After that, the G-DQN (Grouping-DQN) algorithm is proposed to address the problems that the definition of states in existing studies cannot accurately represent the traffic states and the slow convergence of neural networks. Finally, the performance of the G-DQN algorithm model was compared with the original DQN algorithm model and Advantage Actor-Critic (A2C) algorithm model. The experimental results show that the improved algorithm increased the main indicators in all aspects.

Learning Rates of Deep Nets for Geometrically Strongly Mixing Sequence.

The great success of deep learning poses an urgent challenge to establish the theoretical basis for its working mechanism. Recently, research on the convergence of deep neural networks (DNNs) has made great progress. However, the existing studies are based on the assumption that the samples are independent, which is too strong to be applied to many real-world scenarios. In this brief, we establish a fast learning rate for the empirical risk minimization (ERM) on DNN regression with dependent samples, and the dependence is expressed in terms of geometrically strongly mixing sequence. To the best of our knowledge, this is the first convergence result of DNN methods based on mixing sequences. This result is a natural generalization of the independent sample case.

Self-Lateral Propagation Elevates Synaptic Modifications in Spiking Neural Networks for the Efficient Spatial and Temporal Classification.

The brain's mystery for efficient and intelligent computation hides in the neuronal encoding, functional circuits, and plasticity principles in natural neural networks. However, many plasticity principles have not been fully incorporated into artificial or spiking neural networks (SNNs). Here, we report that incorporating a novel feature of synaptic plasticity found in natural networks, whereby synaptic modifications self-propagate to nearby synapses, named self-lateral propagation (SLP), could further improve the accuracy of SNNs in three benchmark spatial and temporal classification tasks. The SLP contains lateral pre ( SLPpre ) and lateral post ( SLPpost ) synaptic propagation, describing the spread of synaptic modifications among output synapses made by axon collaterals or among converging synapses on the postsynaptic neuron, respectively. The SLP is biologically plausible and can lead to a coordinated synaptic modification within layers that endow higher efficiency without losing much accuracy. Furthermore, the experimental results showed the impressive role of SLP in sharpening the normal distribution of synaptic weights and broadening the more uniform distribution of misclassified samples, which are both considered essential for understanding the learning convergence and network generalization of neural networks.

Convergence of deep ReLU networks

We explore convergence of deep neural networks with the popular ReLU activation function, as the depth of the networks tends to infinity. To this end, we introduce the notion of activation domains and activation matrices of a ReLU network. By replacing applications of the ReLU activation function by multiplications with activation matrices on activation domains, we obtain an explicit expression of the ReLU network. We then identify the convergence of the ReLU networks as convergence of a class of infinite products of matrices. Sufficient and necessary conditions for convergence of these infinite products of matrices are studied. As a result, we establish necessary conditions for ReLU networks to converge that the sequence of weight matrices converges to the identity matrix and the sequence of the bias vectors converges to zero as the depth of ReLU networks increases to infinity. Moreover, we obtain sufficient conditions in terms of the weight matrices and bias vectors at hidden layers for pointwise convergence of deep ReLU networks. These results provide mathematical insights to convergence of deep neural networks. Experiments are conducted to mathematically verify the results and to illustrate their potential usefulness in initialization of deep neural networks.

The convergence of a generalized convex neural network

ABSTRACT The convergence of neural networks is a central area of research that is essential in understanding the universal approximation capability and the structural complexity of these systems. In this study, we investigate a generalized convex incremental iteration method. This iteration method is presented in a more general form than previous studies and is capable of scraping a broader range of weight parameters. Additionally, we systematically demonstrate the convergence rate of the convex iteration. Our findings are also easily extensible to deep convolutional neural networks, which helps to explain their effectiveness in various real-world applications. Furthermore, we provide a discrete statistical perspective to address issues of input data non-compactness and unknowability of the objective function in practical settings. To support our conclusions, we propose two algorithms, namely back propagation and random search, the latter of which can prevent the network from getting stuck in a local minimum during training. Finally, we present results from several regression problems, which indicate that our algorithms exhibit superior performance and are consistent with our theoretical predictions. These results provide a more comprehensive understanding of the convergence of neural networks and its significance in the universal approximation capability of these systems.

Achieving stable convergence of neural networks for estimating acoustic field in uniform ducts

The ability of the neural networks as function approximators can be exploited to solve several governing differential equations. In this work, 1-D Helmholz equation is solved to predict the acoustic pressure in a uniform duct. Solving the Helmholtz equation across a range of frequencies, especially at higher frequencies is challenging as the loss function destabilizes the training process, thereby preventing it from converging to the true solution with the desired accuracy. To overcome this issue, a dynamic learning rate technique is proposed that helps to stabilize the training process and improve overall accuracy of the network. The efficiency of the method is demonstrated by comparing the results with a static learning rate method and the analytical solutions. A good agreement is observed between the predicted solution with dynamic learning rate and the analytical solution up to 2000 Hz. Without dynamic learning rate, the relative errors are observed tobe 2% and 58% at 500 and 2000 Hz, respectively, whereas they reduced to 0.6% and 0.1%, respectively, with the dynamic learning rate at the same frequencies. The proposed dynamic learning rate method is found to be effective for different types of boundary conditions.

Convergence of Discrete-Time Cellular Neural Networks with Application to Image Processing

The paper considers a class of discrete-time cellular neural networks (DT-CNNs) obtained by applying Euler’s discretization scheme to standard CNNs. Let [Formula: see text] be the DT-CNN interconnection matrix which is defined by the feedback cloning template. The paper shows that a DT-CNN is convergent, i.e. each solution tends to an equilibrium point, when [Formula: see text] is symmetric and, in the case where [Formula: see text] is not positive-semidefinite, the step size of Euler’s discretization scheme does not exceed a given bound ([Formula: see text] is the [Formula: see text] unit matrix). It is shown that two relevant properties hold as a consequence of the local and space-invariant interconnecting structure of a DT-CNN, namely: (1) the bound on the step size can be easily estimated via the elements of the DT-CNN feedback cloning template only; (2) the bound is independent of the DT-CNN dimension. These two properties make DT-CNNs very effective in view of computer simulations and for the practical applications to high-dimensional processing tasks. The obtained results are proved via Lyapunov approach and LaSalle’s Invariance Principle in combination with some fundamental inequalities enjoyed by the projection operator on a convex set. The results are compared with previous ones in the literature on the convergence of DT-CNNs and also with those obtained for different neural network models as the Brain-State-in-a-Box model. Finally, the results on convergence are illustrated via the application to some relevant 2D and 1D DT-CNNs for image processing tasks.

Deep Fusion Network Based Sparse View CT Reconstructions for Clinical Diagnostic Scanners.

Sparse view CT scan has the advantage of reducing radiation exposure and scanning time in clinical diagnosis. However, the limited number of x-ray projections can make the reconstruction problem ill posed and result in image artifacts. To tackle the problem, we propose a novel model-based deep fusion network(DFN) satisfying the clinical set-up. It extracts fused features encoded from both the sinogram and the preliminary reconstructed image generated by filtered back projection (FBP) to improve the quality of reconstruction. The preliminary reconstructed image endows fused features with prior knowledge that facilitate the convergence of neural network to high-quality reconstruction images. We design a custom loss for training that enforces the network to learn both the pixel value and the integrity of the tissue structure. A synthetic sparse view breast CT dataset from American Association of Physicists in Medicine(AAPM) is used for training, validation and testing. The qualitative and quantitative evaluations show that the DFN reconstruction algorithm significantly improves in balancing between the image quality and reconstruction speed, hence enables fast and high quality CT reconstruction despite the sparse view limitations.

Reward adaptive wind power tracking control based on deep deterministic policy gradient

Wind power efficiency is an essential factor affecting wind power development, and efficient wind power control methods are the key to improving wind power efficiency. Previous wind power control methods require high internal system expertise in internal systems and struggled to balance the accuracy of the maximum power control and the output stability under high wind speed. Therefore, this paper proposes a reward-adaptive control method for wind power tracking based on Deep Deterministic Policy Gradient (DDPG). The method can use one controller to simultaneously control the generator torque and pitch angle in various operating conditions following the system state and the designed flag of operating conditions, thus enabling efficient tracking of the maximum power generation, and stabilizing the output power under high wind speed. Moreover, this paper proposes an internal and external reward algorithm that integrates the Intrinsic Curiosity Module (ICM) and the Actor–Critic (AC) architecture, which can adaptively calculate the reward of DDPG, thereby effectively resolving the sparse reward problem, accelerating the convergence of neural network, and improving the learning effect. From the simulation, the control method proposed in this paper can effectively improve the power generation efficiency under turbulent wind speed, reduce the pitch angle variation by about 30%, and improve the power tracking accuracy by more than 70%.

Fast Convergence in Learning Two-Layer Neural Networks with Separable Data

Fast Convergence in Learning Two-Layer Neural Networks with Separable Data

An AI-Enhanced Strategy of Service Offloading for IoV in Mobile Edge Computing

A full connected world is expected to be introduced in the sixth generation mobile network (6G). As a typical fully connected scenario, the internet of vehicle (IoV) enables intelligent vehicle operations via artificial intelligence (AI) and edge computing technologies. Thus, integrating intelligence into edge computing is, no doubt, a promising development trend. In the future of vehicular networks, a massive variety of services need powerful computing resources and higher quality of service (QoS). Existing computing resources are insufficient to match those increasing requirements. Most works on this problem focused on finding the power-delay’s trade-off, ignoring QoS and stable load balance. In this study, we found that the computing power and redundancy of vehicles’ in IoV is increasing. So, those redundant computing resources are possible to be used to solve the shortage of computing resource. CNN is a typical AI technique. This technology is very suitable for solving the problems in this article. So, we adopted CNN technique of AI to design and algorithm of convolutional long short-term memory (CN_LSTM) based traffic prediction (ACLBTP). ACLBTP was designed to gain the predicted number of vehicles belonging to the edge node. Secondly, according to the problem of insufficient computing resources on remote servers, we found that a large amount of redundant computing resources exist in edge nodes. So, we used edge computing technique to solve the problem of insufficient computing resources on remote servers. ASOBCL was designed to distribute computing tasks to edge nodes. Meanwhile, an intelligent service offloading framework was provided in this article. Based on the framework, an algorithm of improved gradient descent (AIGD) was created to accelerate the speed of iteration. So, the ACLBTP’s convergence of convolutional neural network (CNN) based on AIGD was able to be accelerated too. In ASOBCL, a sorting technique was adopted to speed up the offloading work. Simulation results demonstrated the fact that the prediction strategy designed in this paper had high accuracy. The low offloading time and maintaining stable load balance were gained via running ASOBCL. Low offloading time means short response time. Additionally, the QoS was guaranteed. So, these strategies designed in this paper were effective and valuable.

Optimization of Levenberg Marquardt Algorithm Applied to Nonlinear Systems

As science and technology advance, industrial manufacturing processes get more complicated. Back Propagation Neural Network (BPNN) convergence is comparatively slower for processing nonlinear systems. The nonlinear system used in this study to evaluate the optimization of BPNN based on the LM algorithm proved the algorithm’s efficacy through a MATLAB simulation analysis. This paper examined the application impact of the enhanced approach using the Continuous stirred tank reactor (CSTR) control system as an example. The study’s findings demonstrate that the LM optimization algorithm’s identification error exceeds 10-5. The research’s suggested control approach for reactant concentration CA in CSTR systems provides a better tracking effect and a stronger anti-interference capacity. Compared to the PI control method, the overall control effect is superior. As a result, the optimization model for nonlinear systems has a greatly improved processing accuracy. With some data support for the accuracy study of neural network models and the application of nonlinear systems, the suggested LM-BP optimization algorithm is evidently more appropriate for nonlinear systems.

An adaptive dynamic programming in cooperative target tracking for energy acquisition in wireless sensor networks

In order to further study the application of energy acquisition model, based on the relevant theory of adaptive dynamic programming, the wireless sensor is tracked by modifying the energy acquisition model, and the relevant data and parameters are quantitatively analyzed according to the test results. Finally, the parameters of different samples are tracked and identified. The results show that the evaluation analysis curve with four different parameters can be divided into two stages according to the different variation trends: fluctuation stage and stable stage. It indicates that the increase of time can effectively improve the stability corresponding to the convergence of neural network. The trajectory under the three different parameter states shows a general trend of rapid decline at first and then tends to be stable. There is a certain disturbance in the rapid decline of the curve, but when the time is between 1 and 2, it corresponds to the drop of the curve, and the curve gradually flattens after rapid decline. Parameters in the adaptive dynamic programming model are shown as follows: parameter W decreases slowly to the lowest point and then rises slowly with the increase of iteration steps; The proportion of parameter W1 in the model shows a trend of slow increase. Relevant research can provide theoretical basis for the application of energy acquisition model based on adaptive dynamic programming and analysis in other fields.

Optimal prescribed performance tracking control of nonlinear motor driven systems via adaptive dynamic programming

AbstractAlthough optimal regulation problem has been well studied, resolving optimal tracking control via adaptive dynamic programming (ADP) has not been completely resolved, particularly for nonlinear uncertain systems. In this paper, an online adaptive learning method is developed to realize the optimal tracking control design for nonlinear motor driven systems (NMDSs), which adopts the concept of ADP, unknown system dynamic estimator (USDE), and prescribed performance function (PPF). To this end, the USDE in a simple form is first proposed to address the NMDSs with bounded disturbances. Then, based on the estimated unknown dynamics, we define an optimal cost function and derive the optimal tracking control. The derived optimal tracking control is divided into two parts, that is, steady‐state control and optimal feedback control. The steady‐state control can be obtained with the tracking commands directly. The optimal feedback control can be obtained via the concept of ADP based on the PPF; this contributes to improving the convergence of critic neural network (CNN) weights and tracking accuracy of NMDSs. Simulations are provided to display the feasibility of the designed control method.