Convergence Of Neural Network Research Articles

Self-Lateral Propagation Elevates Synaptic Modifications in Spiking Neural Networks for the Efficient Spatial and Temporal Classification.

The brain's mystery for efficient and intelligent computation hides in the neuronal encoding, functional circuits, and plasticity principles in natural neural networks. However, many plasticity principles have not been fully incorporated into artificial or spiking neural networks (SNNs). Here, we report that incorporating a novel feature of synaptic plasticity found in natural networks, whereby synaptic modifications self-propagate to nearby synapses, named self-lateral propagation (SLP), could further improve the accuracy of SNNs in three benchmark spatial and temporal classification tasks. The SLP contains lateral pre ( SLPpre ) and lateral post ( SLPpost ) synaptic propagation, describing the spread of synaptic modifications among output synapses made by axon collaterals or among converging synapses on the postsynaptic neuron, respectively. The SLP is biologically plausible and can lead to a coordinated synaptic modification within layers that endow higher efficiency without losing much accuracy. Furthermore, the experimental results showed the impressive role of SLP in sharpening the normal distribution of synaptic weights and broadening the more uniform distribution of misclassified samples, which are both considered essential for understanding the learning convergence and network generalization of neural networks.

Uniform Convergence of Deep Neural Networks With Lipschitz Continuous Activation Functions and Variable Widths

Uniform Convergence of Deep Neural Networks With Lipschitz Continuous Activation Functions and Variable Widths

Activation function optimization scheme for image classification

Activation function has a significant impact on the dynamics, convergence, and performance of deep neural networks. The search for a consistent and high-performing activation function has always been a pursuit during deep learning model development. Existing state-of-the-art activation functions are manually designed with human expertise except for Swish. Swish was developed using a reinforcement learning-based search strategy. In this study, we propose an evolutionary approach for optimizing activation functions specifically for image classification tasks, aiming to discover functions that outperform current state-of-the-art options. Through this optimization framework, we obtain a series of high-performing activation functions denoted as Exponential Error Linear Unit (EELU). The developed activation functions are evaluated for image classification tasks from two perspectives: (1) five state-of-the-art neural network architectures, such as ResNet50, AlexNet, VGG16, MobileNet, and Compact Convolutional Transformer, which cover computationally heavy to light neural networks, and (2) eight standard datasets, including CIFAR10, Imagenette, MNIST, Fashion MNIST, Beans, Colorectal Histology, CottonWeedID15, and TinyImageNet which cover from typical machine vision benchmark, agricultural image applications to medical image applications. Finally, we statistically investigate the generalization of the resultant activation functions developed through the optimization scheme. With a Friedman test, we conclude that the optimization scheme is able to generate activation functions that outperform the existing standard ones in 92.8% cases among 28 different cases studied, and −x⋅erf(e−x) is found to be the best activation function for image classification generated by the optimization scheme.

Fault Diagnosis of Distributed Energy Distribution Network Based on PSO-BP

With the increasing scale of distribution network at distribution time, its complexity grows geometrically, and its fault diagnosis becomes more and more difficult. Aiming at the slow convergence and low accuracy of traditional backpropagation neural network in dealing with single-phase ground faults, the study proposes a backpropagation neural network based on improved particle swarm optimization. The model optimizes the weights and acceleration constants of the particle swarm algorithm by introducing dynamic coefficients to enhance its global and local optimization seeking ability. It is also applied in optimizing the parameters of backpropagation neural network and constructing the routing model and ranging model for fault diagnosis about distributed energy distribution network. The simulation results revealed that the maximum absolute error of the improved method is 0.08. While the maximum absolute errors of the traditional backpropagation neural network and the particle swarm optimized backpropagation neural network were 0.65 and 0.10, respectively. The fluctuation of the relative errors of the research method was small under different ranges of measurements. At 8.0 km, the minimum relative error was 0.39% and the maximum relative error was 2.81%. The results show that the improved method proposed in the study significantly improves the accuracy and stability of fault diagnosis and localization in distribution networks and is applicable to complex distribution network environments. The method has high training efficiency and fault detection capability and provides an effective tool for distribution network fault management.

Convergence analysis of a novel high order networks model based on entropy error function

It is generally known that the error function is one of the key factors that determine the convergence, stability and generalization ability of neural networks. For most feedforward neural networks, the squared error function is usually chosen as the error function to train the network. However, networks based on the squared error function can lead to slow convergence and easily fall into local optimum in the actual training process. Recent studies have found that, compared to the squared error function, the gradient method based on the entropy error function measures the difference between the probability distribution of the model output and the probability distribution of the true labels during the iterative process, which can be more able to handle the uncertainty in the classification problem, less likely to fall into a local optimum and can learn to converge more rapidly. In this paper, we propose a batch gradient method for Sigma-Pi-Sigma neural networks based on the entropy error function and rigorously demonstrate the weak and strong convergence of the new algorithm in the batch input mode. Finally, the theoretical results and effectiveness of the algorithm are verified by simulation.

Learning-based robust output tracking control for unknown discrete-time nonlinear systems with dynamic uncertainty

Although numerous approximate dynamic programming (ADP) methods have been proposed and applied to robust control problems, research on the use of ADP methods for robust tracking control in nonlinear systems remains limited, especially for discrete time (DT) nonlinear systems. In this paper, we propose an efficient robust learning-based output tracking control method, referred to as r-LTC, for a class of unknown DT nonlinear systems with dynamic uncertainty. The objectives are achieved by: 1) using a system identification technique based on network-type coefficients Auto-Regressive with eXogenous variables (ARX) model to identify completely unknown systems; 2) reformulating the identified model into a state space model in terms of deviations from the reference trajectory, such that the robust tracking problem can be converted into a robust regulation problem; 3) transforming the robust regulation problem into an equivalent optimal regulation problem for a nominal system with a modified utility function; 4) alternately implementing the performance evaluation and control policy update through two neural networks (NNs) to numerically solve DT Hamilton-Jacobi-Bellman (HJB) equation, such that the approximated optimal feedback control inputs for nominal system can be obtained, allowing for realization of all unknown bounded uncertainties. The proposed method does not require a state observer and any steady state knowledge, only the measurable system input and output data are utilized. The convergence of NNs’ approximate weights and stability of the closed-loop uncertain systems are rigorously proven using the Lyapunov method. Lastly, a case study is performed using real data collected from the Shenzhen Metro Line 12 tunneling project to examine the performance of the proposed r-LTC algorithm in terms of the set value tracking accuracy, control system stability, and robustness against different types of disturbances.

Gradient-based adaptive neural network technique for two-dimensional local fractional elliptic PDEs

This paper introduces gradient-based adaptive neural networks to solve local fractional elliptic partial differential equations. The impact of physics-informed neural networks helps to approximate elliptic partial differential equations governed by the physical process. The proposed technique employs learning the behaviour of complex systems based on input-output data, and automatic differentiation ensures accurate computation of gradient. The method computes the singularity-embedded local fractional partial derivative model on a Hausdorff metric, which otherwise halts the computation by available approximating numerical methods. This is possible because the new network is capable of updating the weight associated with loss terms depending on the solution domain and requirement of solution behaviour. The semi-positive definite character of the neural tangent kernel achieves the convergence of gradient-based adaptive neural networks. The importance of hyperparameters, namely the number of neurons and the learning rate, is shown by considering a stationary anomalous diffusion-convection model on a rectangular domain. The proposed method showcases the network’s ability to approximate solutions of various local fractional elliptic partial differential equations with varying fractal parameters.

Convergence and Recovery Guarantees of Unsupervised Neural Networks for Inverse Problems

Convergence and Recovery Guarantees of Unsupervised Neural Networks for Inverse Problems

Recurrent Neural Network (RNN) Inference Cloud Computing

The purpose of this technical article is to investigate the convergence of Recurrent Neural Networks (RNNs) with Cloud Computing considering the potential benefits and challenges that may be encountered in bringing these two systems together. Recurrent Neural Networks (RNNs) constitute a particular type of artificial neural network that is capable of processing data sequentially in such a way that it becomes well-suited for applications that involve handling time series or language processing. On the other hand, cloud computing provides computer infrastructures on a scalable basis as well as flexible computation services that are accessible whenever they are needed. This paper targets making RNN applications more efficient and faster with the use of RNN libraries on cloud computing platforms through which the cloud services can be used for things like data processing power and storage. By combining RNNs with Cloud Computing, this paper aims to enhance the efficiency and performance of RNN-based applications by leveraging the cloud's computational power and storage capabilities. The study investigates the impact of deploying RNN inference tasks in the cloud environment, analyzing factors such as latency, cost-effectiveness, and scalability.

Investigating the Evolving Landscape of Deepfake Technology: Generative AI's Role in it's Generation and Detection

The world of artificial intelligence is constantly changing, with Generative AI and Large Language Models (LLMs) leading the way in bringing new technological advancements. This paper offers a detailed look at these groundbreaking technologies and how they are shaping the digital world today. We explore the technical aspects of Generative AI and LLMs, explain their unique features, and compare them to traditional AI models.One of the key focuses of our research is the growing issue of DeepFakes—artificial intelligence-generated media that presents a significant challenge in verifying content. We conduct a thorough examination of few deepfake detection techniques out of which we will be implementing and analyzing one of them. Our research implements a framework for Deep Fake Image Detection. The suggested solution utilizes a RESNET-50(Residual Network with 50 layers) and MTCNN (Multi-task Cascaded Convolutional Networks) models for detecting whether the images are real or fake. This study conducts the Hypothesis testing for the proposed solution taking in consideration that the current Deepfake detection algorithms are less effective in detecting highly realistic Deepfakes compared to less sophisticated manipulations. By investigating the convergence of deep learning, neural networks, and sophisticated algorithms, we set the stage for advancements in AI-based content verification.

Can neural networks benefit from objectives that encourage iterative convergent computations? A case study of ResNets and object classification.

Recent work has suggested that feedforward residual neural networks (ResNets) approximate iterative recurrent computations. Iterative computations are useful in many domains, so they might provide good solutions for neural networks to learn. However, principled methods for measuring and manipulating iterative convergence in neural networks remain lacking. Here we address this gap by 1) quantifying the degree to which ResNets learn iterative solutions and 2) introducing a regularization approach that encourages the learning of iterative solutions. Iterative methods are characterized by two properties: iteration and convergence. To quantify these properties, we define three indices of iterative convergence. Consistent with previous work, we show that, even though ResNets can express iterative solutions, they do not learn them when trained conventionally on computer-vision tasks. We then introduce regularizations to encourage iterative convergent computation and test whether this provides a useful inductive bias. To make the networks more iterative, we manipulate the degree of weight sharing across layers using soft gradient coupling. This new method provides a form of recurrence regularization and can interpolate smoothly between an ordinary ResNet and a "recurrent" ResNet (i.e., one that uses identical weights across layers and thus could be physically implemented with a recurrent network computing the successive stages iteratively across time). To make the networks more convergent we impose a Lipschitz constraint on the residual functions using spectral normalization. The three indices of iterative convergence reveal that the gradient coupling and the Lipschitz constraint succeed at making the networks iterative and convergent, respectively. To showcase the practicality of our approach, we study how iterative convergence impacts generalization on standard visual recognition tasks (MNIST, CIFAR-10, CIFAR-100) or challenging recognition tasks with partial occlusions (Digitclutter). We find that iterative convergent computation, in these tasks, does not provide a useful inductive bias for ResNets. Importantly, our approach may be useful for investigating other network architectures and tasks as well and we hope that our study provides a useful starting point for investigating the broader question of whether iterative convergence can help neural networks in their generalization.

A modified U-Net convolutional neural network for segmenting periprostatic adipose tissue based on contour feature learning

ObjectiveThis study trains a U-shaped fully convolutional neural network (U-Net) model based on peripheral contour measures to achieve rapid, accurate, automated identification and segmentation of periprostatic adipose tissue (PPAT). MethodsCurrently, no studies are using deep learning methods to discriminate and segment periprostatic adipose tissue. This paper proposes a novel and modified, U-shaped convolutional neural network contour control points on a small number of datasets of MRI T2W images of PPAT combined with its gradient images as a feature learning method to reduce feature ambiguity caused by the differences in PPAT contours of different patients. This paper adopts a supervised learning method on the labeled dataset, combining the probability and spatial distribution of control points, and proposes a weighted loss function to optimize the neural network's convergence speed and detection performance. Based on high-precision detection of control points, this paper uses a convex curve fitting to obtain the final PPAT contour. The imaging segmentation results were compared with those of a fully convolutional network (FCN), U-Net, and semantic segmentation convolutional network (SegNet) on three evaluation metrics: Dice similarity coefficient (DSC), Hausdorff distance (HD), and intersection over union ratio (IoU). ResultsCropped images with a 270 × 270-pixel matrix had DSC, HD, and IoU values of 70.1%, 27 mm, and 56.1%, respectively; downscaled images with a 256 × 256-pixel matrix had 68.7%, 26.7 mm, and 54.1%. A U-Net network based on peripheral contour characteristics predicted the complete periprostatic adipose tissue contours on T2W images at different levels. FCN, U-Net, and SegNet could not completely predict them. ConclusionThis U-Net convolutional neural network based on peripheral contour features can identify and segment periprostatic adipose tissue quite well. Cropped images with a 270 × 270-pixel matrix are more appropriate for use with the U-Net convolutional neural network based on contour features; reducing the resolution of the original image will lower the accuracy of the U-Net convolutional neural network. FCN and SegNet are not appropriate for identifying PPAT on T2 sequence MR images. Our method can automatically segment PPAT rapidly and accurately, laying a foundation for PPAT image analysis.

Empowering robust biometric authentication: The fusion of deep learning and security image analysis

Many societal entities now have more excellent standards for the efficacy and dependability of identification systems due to the ongoing advancement of computer technology. Traditional identification methods, such as keys and smart cards, have been supplanted by biometric systems in highly secure environments. This research presents a smart computational method for automatically authenticating fingerprints for identity (ID) verification and personal identification. Compared to more traditional machine learning algorithms, the results from applying Deep learning (DL) in areas like computer vision, image identification, robotics, and voice processing have generally been positive. Due to their capacity to analyse big data size and deal with fluctuations in biometric data (such as ageing or expression problems), DL has been heavily used by the artificial intelligence research community. Several biometric systems have succeeded with automatic feature extraction employing deep learning approaches like Convolutional Neural Networks (CNNs). In this research, we provide a biometric process that uses convolutional neural networks. This work introduces a deep learning-based biometric identification system that uses Monte Carlo Dropout (MC Dropout). Combining these two systems makes the authentication process more secure and dependable. Fingerprint image enhancement techniques involve the application of Gabor filters and structure-adaptive anisotropic filters, which have proven to be effective in enhancing the clarity and distinctiveness of fingerprint patterns. To improve the efficiency of deep learning models, this work proposes the Inception-Augmentation GAN (IAGAN) model for data augmentation. The study adds to security development by integrating novel biometric identification and authentication approaches with cutting-edge neural network technology. In this research, we provide a new activation function to speed up the convergence of deep neural networks. The results of 99.6% on Gabor filters and 99.8% on the structure-adaptive anisotropic filter with GACNN with MCD show that deep neural networks can excel over competing approaches with enough training data.

Optimization Control of Adaptive Traffic Signal with Deep Reinforcement Learning

The optimization and control of traffic signals is very important for logistics transportation. It not only improves the operational efficiency and safety of road traffic, but also conforms to the direction of the intelligent, green, and sustainable development of modern cities. In order to improve the optimization effect of traffic signal control, this paper proposes a traffic signal optimization method based on deep reinforcement learning and Simulation of Urban Mobility (SUMO) software for urban traffic scenarios. The intersection training scenario was established using SUMO micro traffic simulation software, and the maximum vehicle queue length and vehicle queue time were selected as performance evaluation indicators. In order to be more relevant to the real environment, the experiment uses Weibull distribution to simulate vehicle generation. Since deep reinforcement learning takes into account both perceptual and decision-making capabilities, this study proposes a traffic signal optimization control model based on the deep reinforcement learning Deep Q Network (DQN) algorithm by considering the realism and complexity of traffic intersections, and first uses the DQN algorithm to train the model in a training scenario. After that, the G-DQN (Grouping-DQN) algorithm is proposed to address the problems that the definition of states in existing studies cannot accurately represent the traffic states and the slow convergence of neural networks. Finally, the performance of the G-DQN algorithm model was compared with the original DQN algorithm model and Advantage Actor-Critic (A2C) algorithm model. The experimental results show that the improved algorithm increased the main indicators in all aspects.

Learning Rates of Deep Nets for Geometrically Strongly Mixing Sequence.

The great success of deep learning poses an urgent challenge to establish the theoretical basis for its working mechanism. Recently, research on the convergence of deep neural networks (DNNs) has made great progress. However, the existing studies are based on the assumption that the samples are independent, which is too strong to be applied to many real-world scenarios. In this brief, we establish a fast learning rate for the empirical risk minimization (ERM) on DNN regression with dependent samples, and the dependence is expressed in terms of geometrically strongly mixing sequence. To the best of our knowledge, this is the first convergence result of DNN methods based on mixing sequences. This result is a natural generalization of the independent sample case.

Convergence of deep ReLU networks

We explore convergence of deep neural networks with the popular ReLU activation function, as the depth of the networks tends to infinity. To this end, we introduce the notion of activation domains and activation matrices of a ReLU network. By replacing applications of the ReLU activation function by multiplications with activation matrices on activation domains, we obtain an explicit expression of the ReLU network. We then identify the convergence of the ReLU networks as convergence of a class of infinite products of matrices. Sufficient and necessary conditions for convergence of these infinite products of matrices are studied. As a result, we establish necessary conditions for ReLU networks to converge that the sequence of weight matrices converges to the identity matrix and the sequence of the bias vectors converges to zero as the depth of ReLU networks increases to infinity. Moreover, we obtain sufficient conditions in terms of the weight matrices and bias vectors at hidden layers for pointwise convergence of deep ReLU networks. These results provide mathematical insights to convergence of deep neural networks. Experiments are conducted to mathematically verify the results and to illustrate their potential usefulness in initialization of deep neural networks.

The convergence of a generalized convex neural network

ABSTRACT The convergence of neural networks is a central area of research that is essential in understanding the universal approximation capability and the structural complexity of these systems. In this study, we investigate a generalized convex incremental iteration method. This iteration method is presented in a more general form than previous studies and is capable of scraping a broader range of weight parameters. Additionally, we systematically demonstrate the convergence rate of the convex iteration. Our findings are also easily extensible to deep convolutional neural networks, which helps to explain their effectiveness in various real-world applications. Furthermore, we provide a discrete statistical perspective to address issues of input data non-compactness and unknowability of the objective function in practical settings. To support our conclusions, we propose two algorithms, namely back propagation and random search, the latter of which can prevent the network from getting stuck in a local minimum during training. Finally, we present results from several regression problems, which indicate that our algorithms exhibit superior performance and are consistent with our theoretical predictions. These results provide a more comprehensive understanding of the convergence of neural networks and its significance in the universal approximation capability of these systems.

Achieving stable convergence of neural networks for estimating acoustic field in uniform ducts

The ability of the neural networks as function approximators can be exploited to solve several governing differential equations. In this work, 1-D Helmholz equation is solved to predict the acoustic pressure in a uniform duct. Solving the Helmholtz equation across a range of frequencies, especially at higher frequencies is challenging as the loss function destabilizes the training process, thereby preventing it from converging to the true solution with the desired accuracy. To overcome this issue, a dynamic learning rate technique is proposed that helps to stabilize the training process and improve overall accuracy of the network. The efficiency of the method is demonstrated by comparing the results with a static learning rate method and the analytical solutions. A good agreement is observed between the predicted solution with dynamic learning rate and the analytical solution up to 2000 Hz. Without dynamic learning rate, the relative errors are observed tobe 2% and 58% at 500 and 2000 Hz, respectively, whereas they reduced to 0.6% and 0.1%, respectively, with the dynamic learning rate at the same frequencies. The proposed dynamic learning rate method is found to be effective for different types of boundary conditions.

Convergence of Discrete-Time Cellular Neural Networks with Application to Image Processing

The paper considers a class of discrete-time cellular neural networks (DT-CNNs) obtained by applying Euler’s discretization scheme to standard CNNs. Let [Formula: see text] be the DT-CNN interconnection matrix which is defined by the feedback cloning template. The paper shows that a DT-CNN is convergent, i.e. each solution tends to an equilibrium point, when [Formula: see text] is symmetric and, in the case where [Formula: see text] is not positive-semidefinite, the step size of Euler’s discretization scheme does not exceed a given bound ([Formula: see text] is the [Formula: see text] unit matrix). It is shown that two relevant properties hold as a consequence of the local and space-invariant interconnecting structure of a DT-CNN, namely: (1) the bound on the step size can be easily estimated via the elements of the DT-CNN feedback cloning template only; (2) the bound is independent of the DT-CNN dimension. These two properties make DT-CNNs very effective in view of computer simulations and for the practical applications to high-dimensional processing tasks. The obtained results are proved via Lyapunov approach and LaSalle’s Invariance Principle in combination with some fundamental inequalities enjoyed by the projection operator on a convex set. The results are compared with previous ones in the literature on the convergence of DT-CNNs and also with those obtained for different neural network models as the Brain-State-in-a-Box model. Finally, the results on convergence are illustrated via the application to some relevant 2D and 1D DT-CNNs for image processing tasks.

Deep Fusion Network Based Sparse View CT Reconstructions for Clinical Diagnostic Scanners.

Sparse view CT scan has the advantage of reducing radiation exposure and scanning time in clinical diagnosis. However, the limited number of x-ray projections can make the reconstruction problem ill posed and result in image artifacts. To tackle the problem, we propose a novel model-based deep fusion network(DFN) satisfying the clinical set-up. It extracts fused features encoded from both the sinogram and the preliminary reconstructed image generated by filtered back projection (FBP) to improve the quality of reconstruction. The preliminary reconstructed image endows fused features with prior knowledge that facilitate the convergence of neural network to high-quality reconstruction images. We design a custom loss for training that enforces the network to learn both the pixel value and the integrity of the tissue structure. A synthetic sparse view breast CT dataset from American Association of Physicists in Medicine(AAPM) is used for training, validation and testing. The qualitative and quantitative evaluations show that the DFN reconstruction algorithm significantly improves in balancing between the image quality and reconstruction speed, hence enables fast and high quality CT reconstruction despite the sparse view limitations.