Implementation Of Neural Networks Research Articles

Prescribed Performance Control for Solving Time-Varying Underdetermined Linear Systems With Bounds on States and Their Derivatives

This article addresses the problem of finding online solutions to time-varying underdetermined linear systems with limits on states and their derivatives through a novel zeroing neural network (ZNN) implementation. The proposed model combines zeroing dynamics with user-prescribed performance constraints to ensure that the system achieves desired transient and steady-state behavior. The novelty of this approach lies in a nonlinear invertible mapping that transforms the constrained system to an unconstrained one so that the resulting error remains subsequently bounded by the performance function for all time. In particular, two different ZNN models are synthesized that rely on an exponentially convergent and finite-time convergent performance function to drive the residual error to convergence. The effectiveness of the proposed models is verified through redundancy resolution in path-tracking problems in robotics. A detailed performance comparison study with two leading alternative designs is also undertaken to further illustrate the merits of the proposed schemes.

Comparative Analysis of the Implementation of Support Vector Machines and Long Short-Term Memory Artificial Neural Networks in Municipal Solid Waste Management Models in Megacities

The development of methodologies to support decision-making in municipal solid waste (MSW) management processes is of great interest for municipal administrations. Artificial intelligence (AI) techniques provide multiple tools for designing algorithms to objectively analyze data while creating highly precise models. Support vector machines and neuronal networks are formed by AI applications offering optimization solutions at different managing stages. In this paper, an implementation and comparison of the results obtained by two AI methods on a solid waste management problem is shown. Support vector machine (SVM) and long short-term memory (LSTM) network techniques have been used. The implementation of LSTM took into account different configurations, temporal filtering and annual calculations of solid waste collection periods. Results show that the SVM method properly fits selected data and yields consistent regression curves, even with very limited training data, leading to more accurate results than those obtained by the LSTM method.

Condition monitoring of wind turbines with the implementation of spatio-temporal graph neural network

Condition monitoring of wind turbines is critical to ensure their long-term stable operation. With the benefit of deep learning techniques, WTs’ health status information can be mined more fully from supervisory control and data acquisition data. However, these deep learning-based condition monitoring methods have the following limitations. (1) They only can process regularly structured data, such as pictures, rather than general domains. (2) The spatial properties of wind turbines multi-sensor networks, i.e., connectivity and globality, are neglected. To overcome the above limitations, a new condition monitoring network named spatio-temporal graph neural network is proposed in this paper. First, the missing value supplement and the selection of variables with maximal information coefficient are applied. Meanwhile, the top-k nearest neighbors is employed to construct graphs. Then, a spatio-temporal block is established based on graph convolution networks and gated recurrent unit. By stacking multiple spatio-temporal blocks, the monitoring variables are estimated by feeding the learned features to the last prediction layer. Lastly, the proposed spatio-temporal graph neural network is validated using real wind farm supervisory control and data acquisition data. The experimental results indicate that the proposed method can detect the early abnormal operation efficiently and is superior to some existing methods, which can promote the utilization of renewable energy.

Application of supervised machine learning to predict the enhanced gas recovery by CO2 injection in shale gas reservoirs

The technique of Enhanced Gas Recovery by CO2 injection (CO2-EGR) into shale reservoirs has brought increasing attention in the recent decade. CO2-EGR is a complex geophysical process that is controlled by several parameters of shale properties and engineering design. Nevertheless, more challenges arise when simulating and predicting CO2/CH4 displacement within the complex pore systems of shales. Therefore, the petroleum industry is in need of developing a cost-effective tool/approach to evaluate the potential of applying CO2 injection to shale reservoirs. In recent years, machine learning applications have gained enormous interest due to their high-speed performance in handling complex data and efficiently solving practical problems. Thus, this work proposes a solution by developing a supervised machine learning (ML) based model to preliminary evaluate CO2-EGR efficiency. Data used for this work was drawn across a wide range of simulation sensitivity studies and experimental investigations. In this work, linear regression and artificial neural networks (ANNs) implementations were considered for predicting the incremental enhanced CH4. Based on the model performance in training and validation sets, our accuracy comparison showed that (ANNs) algorithms gave 15% higher accuracy in predicting the enhanced CH4 compared to the linear regression model. To ensure the model is more generalizable, the size of hidden layers of ANNs was adjusted to improve the generalization ability of ANNs model. Among ANNs models presented, ANNs of 100 hidden layer size gave the best predictive performance with the coefficient of determination (R2) of 0.78 compared to the linear regression model with R2 of 0.68. Our developed ML-based model presents a powerful, reliable and cost-effective tool which can accurately predict the incremental enhanced CH4 by CO2 injection in shale gas reservoirs.

Multi-input parallel graph neural network for semi-supervised rolling bearing fault diagnosis

Rolling bearing fault diagnosis is the key technology to ensure the reliable, efficient and sustainable operation of rotating machinery. Many fault diagnosis methods have been proposed based on vibration signal analysis from the perspective of data-driven analytics. However, these methods normally take signals of multiple sensors as a whole for feature extraction without considering the relationship among samples. This drawback leads to insufficient feature mining, thereby affecting the accuracy of fault diagnosis. Moreover, these methods need large numbers of labeled samples to achieve high diagnosis accuracy, which requires extensive human labor and is impractical in many real-world applications. To address these issues, a semi-supervised rolling bearing fault diagnosis method based on multi-input parallel graph neural network is proposed in this paper. In the proposed model, signals of multiple sensors are treated separately; thus, features will be extracted parallelly in a more sufficient way. Then, signals of each sensor are constructed into a graph based on limited-radius nearest neighbor, which will add extra relationship information to aid in fault diagnosis. In addition, with the implementation of graph convolutional neural network, the proposed method is able to achieve a more accurate diagnosis than the comparison methods in the case of few labeled data. Finally, the proposed model is evaluated on rolling bearing dataset provided by Case Western Reserve University. Compared with some classical fault diagnosis methods, the proposed model can improve the diagnosis accuracy up to more than 99% even when the proportion of training samples is only 20%.

End-to-End Implementation of a Convolutional Neural Network on a 3D-Integrated Image Sensor with Macropixel Array.

Three-dimensional-integrated focal-plane array image processor chips offer new opportunities to implement highly parallelised computer vision algorithms directly inside sensors. Neural networks in particular can perform highly complex machine vision tasks, and therefore their efficient implementation in such imagers are of significant interest. However, studies with existing pixel-processor array chips have focused on the implementation of a subset of neural network components-notably convolutional kernels-on pixel processor arrays. In this work, we implement a continuous end-to-end pipeline for a convolutional neural network from the digitisation of incoming photons to the output prediction vector on a macropixel processor array chip (where a single processor acts on group of pixels). Our implementation performs inference at a rate between 265 and 309 frames per second, directly inside of the sensor, by exploiting the different levels of parallelism available.

High‐Temporal‐Resolution Characterization Reveals Outstanding Random Telegraph Noise and the Origin of Dielectric Breakdown in h‐BN Memristors

AbstractMemristor‐based electronic memory have recently started commercialization, although its market size is small (~0.5%). Multiple studies claim their potential for hardware implementation of artificial neural networks, advanced data encryption, and high‐frequency switches for 5G/6G communication. Application aside, the performance and reliability of memristors need to be improved to increase their market size and fit technology standards. Multiple groups propose novel nano‐materials beyond phase‐change, metal‐oxides, and magnetic materials as resistive switching medium (e.g., two‐dimensional, nanowires, perovskites). However, most studies use characterization setups that are blind to critical phenomena in understanding charge transport across the devices. Here an advanced setup with high temporal resolution is used to analyze current noise, dielectric breakdown growth, and ambipolar resistive switching in memristors based on multilayer hexagonal boron nitride (h‐BN), one of the most promising novel nano‐materials for memristive applications. The random telegraph noise in pristine memristors and its evolution as the devices degrade, covering ~7 orders of magnitude in current with consistent observation, is studied. Additionally, an ambipolar switching regime with very low resistance down to 50Ω and its connection with a telegraph behavior with high/low current ratios >100, linked to a thermally‐driven disruption of a metallic nanofilament, is shown.

Simulation and implementation of two-layer oscillatory neural networks for image edge detection: bidirectional and feedforward architectures

The growing number of edge devices in everyday life generates a considerable amount of data that current AI algorithms, like artificial neural networks, cannot handle inside edge devices with limited bandwidth, memory, and energy available. Neuromorphic computing, with low-power oscillatory neural networks (ONNs), is an alternative and attractive solution to solve complex problems at the edge. However, ONN is currently limited with its fully-connected recurrent architecture to solve auto-associative memory problems. In this work, we use an alternative two-layer bidirectional ONN architecture. We introduce a two-layer feedforward ONN architecture to perform image edge detection, using the ONN to replace convolutional filters to scan the image. Using an HNN Matlab emulator and digital ONN design simulations, we report efficient image edge detection from both architectures using various size filters (3 × 3, 5 × 5, and 7 × 7) on black and white images. In contrast, the feedforward architectures can also perform image edge detection on gray scale images. With the digital ONN design, we also assess latency performances and obtain that the bidirectional architecture with a 3 × 3 filter size can perform image edge detection in real-time (camera flow from 25 to 30 images per second) on images with up to 128 × 128 pixels while the feedforward architecture with same 3 × 3 filter size can deal with 170 × 170 pixels, due to its faster computation.

Condition-Based Maintenance of an Anaerobic Reactor Using Artificial Intelligence

This paper proposes a condition-based maintenance system based on artificial intelligence for an online monitoring system of the support bed expansion in a 30-liter pilot-scale inverse fluidized bed reactor (IFBR). The main scope is to achieve a condition-based maintenance strategy using a single-level sensor for a biofilm inverse fluidizing bed as source for virtual sensors. The implementation of an artificial neural network was performed on an embedded electronic system (Raspberry Pi 4), both working together in real time. The signals estimated by the neural network are compared against the signals measured by the hardware sensors and, in case of detecting a failure in the physical measurement system, the artificial intelligence-based system then uses the signal estimated by the artificial neural network to maintain the correct operation of the IFBR. This system uses an artificial neural network to estimate the COD concentration of the effluent and the biogas production flow of a bioreactor, from the measurement of pH, the COD concentration of the influent, the inflow to the bioreactor and the signal coming from each of the conductivity sensors installed inside the reactor, which provide information about support media expansion in a pilot-scale inverse fluidized bed reactor. In addition, a fuzzy PI controller is presented, which was implemented in a Raspberry Pi electronic card, to regulate the COD concentration in the effluent of the bioreactor used as a case study.

Multiclass datasets expand neural network utility: an example on ankle radiographs

PurposeArtificial intelligence in computer vision has been increasingly adapted in clinical application since the implementation of neural networks, potentially providing incremental information beyond the mere detection of pathology. As its algorithmic approach propagates input variation, neural networks could be used to identify and evaluate relevant image features. In this study, we introduce a basic dataset structure and demonstrate a pertaining use case.MethodsA multidimensional classification of ankle x-rays (n = 1493) rating a variety of features including fracture certainty was used to confirm its usability for separating input variations. We trained a customized neural network on the task of fracture detection using a state-of-the-art preprocessing and training protocol. By grouping the radiographs into subsets according to their image features, the influence of selected features on model performance was evaluated via selective training.ResultsThe models trained on our dataset outperformed most comparable models of current literature with an ROC AUC of 0.943. Excluding ankle x-rays with signs of surgery improved fracture classification performance (AUC 0.955), while limiting the training set to only healthy ankles with and without fracture had no consistent effect.ConclusionUsing multiclass datasets and comparing model performance, we were able to demonstrate signs of surgery as a confounding factor, which, following elimination, improved our model. Also eliminating pathologies other than fracture in contrast had no effect on model performance, suggesting a beneficial influence of feature variability for robust model training. Thus, multiclass datasets allow for evaluation of distinct image features, deepening our understanding of pathology imaging.

Effect of Device Mismatches in Differential Oscillatory Neural Networks

Analog implementation of Oscillatory Neural Networks (ONNs) has the potential to implement fast and ultra-low-power computing capabilities. One of the drawbacks of analog implementation is component mismatches which cause desynchronization and instability in ONNs. Emerging devices like memristors and VO2are particularly prone to variations. In this paper, we study the effect of component mismatches on the performance of differential ONNs (DONNs). Mismatches were considered in two main blocks: differential oscillatory neurons and synaptic circuits. To measure DONN tolerance to mismatches in each block, performance was evaluated with mismatches being present separately in each block. Memristor-bridge circuits with four memristors were used as the synaptic circuits. The differential oscillatory neurons were based on VO2-devices. The simulation results showed that DONN performance was more vulnerable to mismatches in the components of the differential oscillatory neurons than to mismatches in the synaptic circuits. DONNs were found to tolerate up to 20% mismatches in the memristance of the synaptic circuits. However, mismatches in the differential oscillatory neurons resulted in non-uniformity of the natural frequencies, causing desynchronization and instability. Simulations showed that 0.5% relative standard deviation (RSD) in natural frequencies can reduce DONN performance dramatically. In addition, sensitivity analyses showed that the high threshold voltage of VO2-devices is the most sensitive parameter for frequency non-uniformity and desynchronization.

Convolutional networks with short-term memory effects

The scale expansion and depth increase of convolutional neural network lead to the increase in the number of devices and chip size, which makes the hardware implementation of neural network more and more difficult. To solve this problem, we design a convolutional neural network with short-term memory to save the number of devices and reduce the complexity of the hardware implementation. Memristor, as a natural nanoscale synapse, is one of the important devices to realize convolutional neural network chip. However, most current memristor-based neural networks use non-volatile memristors. Non-volatile memristors can only store long-term weights because of the stability of resistance, but volatile memristor can store short-term and long-term weights due to its forgetting property. This paper designs a LeNet-5 convolutional neural network by using forgetting memristor bridges composed of volatile memristors. Due to the short-term memory of volatile memristors, 49% of the number of memristors can be effectively saved compared to the neural networks using non-volatile memristors. By writing different weights for the forgetting memristor bridges, a recognition accuracy of about 97% is achieved on the MNIST dataset, and a good recognition accuracy is still achieved at 40,000 images if a recovery signal is given. What is more, in the FASHION-MNIST and ORL datasets, we also achieve 85% and 91% recognition accuracy respectively with the same network.

Optimization of Deep Neural Networks Using DAPP (DNN Acceleration Using Ping-Pong) Approach

Abstract: Deep Neural Networks (DNNs) are one of the leading classification algorithms. Deep Learning has achieved remarkable milestones such as Google-Net and Alpha-Go. They have shown promising results in pattern recognition for images and text, language translation, sound recognition, and many more. DNN has been widely accepted and also employed for pattern matching and image recognition. All these applications are possible as these networks emulate functioning of human brain and hence name “Neural” Network.However, to provide competent results, these millions of neurons in neural networks needs to be trained. For which billions of operations are to be carried out. Training of these many neurons and with these many operations is a time-consuming affair. Hence choice of network and its parameter play an important role both in providing accurate trained network and time taken for training. If the network is deep and has plethora of neurons, the time taken is considerably high as training works sequentially on batches of dataset using Sequential Back-propagation Algorithm.To accelerate the training there are many hardware solutions like use of GPU, FPGA and ASICs. However because of popularity of DNN there is increase in demand in mobile and IoT platform devices. These are resource constrained devices, where power and size of these device, restricts usage and implementation of deep NN (Neural Network).Simulation of DAPP is done on MNIST and CIFAR-10 datasets using System-C. Additionally, this technique has been adapted for multi-core architectures. The design shows a reduction in time by 38% for 3 layers of CNN and 92% for 10 layers of CNN, while maintaining the accuracy of networks. This generic methodology has been implemented for Vanilla RNN and LSTM networks. An improvement of 38% for Vanilla RNN and 40% for LSTM has been demonstrated by this methodology.

Highly-scaled and fully-integrated 3-dimensional ferroelectric transistor array for hardware implementation of neural networks

Hardware-based neural networks (NNs) can provide a significant breakthrough in artificial intelligence applications due to their ability to extract features from unstructured data and learn from them. However, realizing complex NN models remains challenging because different tasks, such as feature extraction and classification, should be performed at different memory elements and arrays. This further increases the required number of memory arrays and chip size. Here, we propose a three-dimensional ferroelectric NAND (3D FeNAND) array for the area-efficient hardware implementation of NNs. Vector-matrix multiplication is successfully demonstrated using the integrated 3D FeNAND arrays, and excellent pattern classification is achieved. By allocating each array of vertical layers in 3D FeNAND as the hidden layer of NN, each layer can be used to perform different tasks, and the classification of color-mixed patterns is achieved. This work provides a practical strategy to realize high-performance and highly efficient NN systems by stacking computation components vertically.

Predictive Maintenance and Fault Monitoring Enabled by Machine Learning: Experimental Analysis of a TA-48 Multistage Centrifugal Plant Compressor

In an increasingly competitive industrial world, the need to adapt to any change at any time has become a major necessity for every industry to remain competitive and survive in their environments. Industries are undergoing rapid and perpetual changes on several levels. Indeed, the latter requires companies to be more reactive and involved in their policies of continuous improvement in order to satisfy their customers and maximize the quantity and quality of production, while keeping the cost of production as low as possible. Reducing downtime is one of the major objectives of these industries of the future. This paper aimed to apply machine learning algorithms on a TA-48 multistage centrifugal compressor for failure prediction and remaining useful life (RUL), i.e., to reduce system downtime using a predictive maintenance (PdM) approach through the adoption of Industry 4.0 approaches. To achieve our goal, we followed the methodology of the predictive maintenance workflow that allows us to explore and process the data for the model training. Thus, a comparative study of different prediction algorithms was carried out to arrive at the final choice, which is based on the implementation of LSTM neural networks. In addition, its performance was improved as the data sets were fed and incremented. Finally, the model was deployed to allow operators to know the failure times of compressors and subsequently ensure minimum downtime rates by making decisions before failures occur.

Trajectory tracking double two-loop adaptive neural network control for a Quadrotor

In this paper, the development and experimental validation of a novel double two-loop nonlinear controller based on adaptive neural networks for a quadrotor are presented. The proposed controller has a two-loop structure: an outer loop for position control and an inner loop for attitude control. Similarly, both position and orientation controllers also have a two-loop design with an adaptive neural network in each inner loop. The output weight matrices of the neural networks are updated online through adaptation laws obtained from a rigorous error convergence analysis. Thus, a training stage is unnecessary prior to the neural network implementation. Additionally, an integral action is included in the controller to cope with constant disturbances. The error convergence analysis guarantees the achievement of the trajectory tracking task and the boundedness of the output weight matrix estimation errors. The proposed scheme is designed such that an accurate knowledge of the quadrotor parameters is not needed. A comparison against the proposed controller and two other well-known schemes is presented. The obtained results showed the functionality of the proposed controller and demonstrated robustness to parametric uncertainty.

Water and wastewater quality prediction: current trends and challenges in the implementation of artificial neural network.

Traditional freshwater supplies have been over-abstracted in the current global problem of water scarcity. Through the analysis of complex experimental and real-time data, to improve the activity of water and wastewater treatment (WWT) systems, an artificial neural network (ANN), a computational model inspired by the human brain, and its variants were created. This review paper focuses on recent trends and advances in modeling and simulating different water and wastewater systems using ANN. This study uses ANN in watershed management, impurity removal from wastewater, and wastewater treatment plants. According to the literature review, ANN can predict nonlinear, linear, and complex systems with high accuracy and well control. Finally, the limitations and future scope of ANNs were discussed. ANN proved itself in the prediction of various water and WWT processes. Still, implementation has practical challenges, which include a lack of data availability, poorly built models, timely updates in developed models, and low repeatability. The use of a proper toolbox, faster computing power, and proper domain knowledge makes the practical implementation of ANN successful. As a result, ANN can build a solid foundation for attracting and motivating investigators to work in this region in the forthcoming.

Implementation of a convolutional neural network using Tensorflow machine learning platform

The generalized algorithm of a typical convolutional neural network realization by means of TensorFlow machine learning library is considered. The peculiarities of the coding implementation of the convolutional neural network in the image recognition problem are analyzed with the example of the MNIST datаset.

Development and Implementation of an ANN Based Flow Law for Numerical Simulations of Thermo-Mechanical Processes at High Temperatures in FEM Software

Numerical methods based on finite element (FE) have proven their efficiency for many years in the thermomechanical simulation of forming processes. Nevertheless, the application of these methods to new materials requires the identification and implementation of constitutive and flow laws within FE codes, which sometimes pose problems, particularly because of the strongly non-linear character of the behavior of these materials. Computational techniques based on machine learning and artificial neural networks are becoming more and more important in the development of these models and help the FE codes to integrate more complex behavior. In this paper, we present the development, implementation and use of an artificial neural network (ANN) based flow law for a GrC15 alloy under high temperature thermomechanical solicitations. The flow law modeling by ANN shows a significant superiority in terms of model prediction quality compared to classical approaches based on widely used Johnson–Cook or Arrhenius models. Once the ANN parameters have been identified on the base of experiments, the implementation of this flow law in a finite element code shows promising results in terms of solution quality and respect of the material behavior.

INVESTGATING POWER METRICS OF NEURAL NETWORKS AFTER PRUNING, QUANTIZATION, DPU IMPLEMENTATION: A COMPARATIVE ANALYSIS

The rapid proliferation of deep learning applications in various fields has highlighted the need for efficient neural network implementations, especially on resource-constrained edge devices. In response to this demand, pruning and quantization have emerged as essential techniques to reduce the computational and memory requirements of neural networks. Additionally, the deployment of dedicated hardware, such as Digital Processing Units (DPUs), has gained momentum for accelerating neural network inference. This paper presents a comprehensive comparative analysis of the power metrics of neural networks after pruning and quantization, with a particular focus on their implement-ation on DPUs. The objective of this research is to investigate the energy efficiency and power consumption of pruned and quantized neural networks when executed on DPU platforms. The trade-offs between model size reduction and inference accuracy, as well as the power efficiency of different DPU architectures are researched. The results reveal insights into the power efficiency of pruned and quantized neural networks on DPU platforms, offering a clear understanding of the benefits and trade-offs associated with these optimization techniques. This research provides a valuable resource for researchers, developers, and practitioners interested in optimizing neural network implementations for power efficiency. The findings contribute to the ongoing effort to make deep learning more accessible and sustainable on edge devices and other power-constrained environments, ultimately enabling a wider range of applications with reduced energy consumption.