Feedforward Network Architecture Research Articles

AI-Powered Neural Network Verification: System Verilog Methodologies for Machine Learning in Hardware

This research focuses on verifying neural network models using System Verilog, with two primary applications: visual edge detection and neuron behavior modeling. In modern chip design, hardware verification plays a crucial role in ensuring that complex neural models perform as expected. A neuron model based on Hubel and Wiesel’s feed-forward network architecture was proposed and tested using integrator and threshold modules implemented in Verilog. The proposed verification methodology employs self-checking test benches, supported by functional coverage and simulation, for comprehensive validation. The results demonstrate efficient verification with high coverage, paving the way for future advancements in hardware neural networks.

Turbofan engine health status prediction with neural network pattern recognition and automated feature engineering

PurposeThis study aims to present the concept of aircraft turbofan engine health status prediction with artificial neural network (ANN) pattern recognition but augmented with automated features engineering (AFE).Design/methodology/approachThe main concept of engine health status prediction was based on three case studies and a validation process. The first two were performed on the engine health status parameters, namely, performance margin and specific fuel consumption margin. The third one was generated and created for the engine performance and safety data, specifically created for the final test. The final validation of the neural network pattern recognition was the validation of the proposed neural network architecture in comparison to the machine learning classification algorithms. All studies were conducted for ANN, which was a two-layer feedforward network architecture with pattern recognition. All case studies and tests were performed for both simple pattern recognition network and network augmented with automated feature engineering (AFE).FindingsThe greatest achievement of this elaboration is the presentation of how on the basis of the real-life engine operational data, the entire process of engine status prediction might be conducted with the application of the neural network pattern recognition process augmented with AFE.Practical implicationsThis research could be implemented into the engine maintenance strategy and planning. Engine health status prediction based on ANN augmented with AFE is an extremely strong tool in aircraft accident and incident prevention.Originality/valueAlthough turbofan engine health status prediction with ANN is not a novel approach, what is absolutely worth emphasizing is the fact that contrary to other publications this research was based on genuine, real engine performance operational data as well as AFE methodology, which makes the entire research very reliable. This is also the reason the prediction results reflect the effect of the real engine wear and deterioration process.

Deep operator networks for bioheat transfer problems with parameterized laser source functions

To numerically characterize optothermal interactions at the laser-tissue interface during tissue imaging or surgery, the bioheat transfer partial differential equation (PDE) needs to be solved with a laser excitation source function. Considering various possible experimental scenarios, the PDEs become parametric and need to be solved for several optical and laser parameters. Classical numerical solvers that are used for performing such simulations are computationally expensive and time consuming. In this work, we use the recent deep operator network (DeepONet) framework for learning the mapping between infinite dimensional function spaces of the parameterized laser source functions and the solutions of the parameterized bioheat transfer PDE. We propose a new neural network architecture, Res-DeepONet, for training a purely physics-informed DeepONet with laser heating functions comprising of Gaussian beam profiles. We also showcase the versatility and robustness of our network architecture within purely data-driven and hybrid training frameworks for applications where labeled ground truth data may be accessible. The proposed neural network architecture exhibits excellent generalization performance on unseen Gaussian source functions with 10× higher accuracy when compared to traditional feedforward network architectures. Furthermore, zero-shot performance studies highlight the exceptional extrapolation capabilities of our network with 50× higher accuracy compared to traditional DeepONet architectures at inference times that are two orders of magnitude faster than classical solvers.

Theoretical Exploration of Flexible Transmitter Model.

Neural network models generally involve two important components, i.e., network architecture and neuron model. Although there are abundant studies about network architectures, only a few neuron models have been developed, such as the MP neuron model developed in 1943 and the spiking neuron model developed in the 1950s. Recently, a new bio-plausible neuron model, flexible transmitter (FT) model (Zhang and Zhou, 2021), has been proposed. It exhibits promising behaviors, particularly on temporal-spatial signals, even when simply embedded into the common feedforward network architecture. This article attempts to understand the properties of the FT network (FTNet) theoretically. Under mild assumptions, we show that: 1) FTNet is a universal approximator; 2) the approximation complexity of FTNet can be exponentially smaller than those of commonly used real-valued neural networks with feedforward/recurrent architectures and is of the same order in the worst case; and 3) any local minimum of FTNet is the global minimum, implying that it is possible to identify global minima by local search algorithms.

Multi-label classification of retinal disease via a novel vision transformer model.

The precise identification of retinal disorders is of utmost importance in the prevention of both temporary and permanent visual impairment. Prior research has yielded encouraging results in the classification of retinal images pertaining to a specific retinal condition. In clinical practice, it is not uncommon for a single patient to present with multiple retinal disorders concurrently. Hence, the task of classifying retinal images into multiple labels remains a significant obstacle for existing methodologies, but its successful accomplishment would yield valuable insights into a diverse array of situations simultaneously. This study presents a novel vision transformer architecture called retinal ViT, which incorporates the self-attention mechanism into the field of medical image analysis. To note that this study supposed to prove that the transformer-based models can achieve competitive performance comparing with the CNN-based models, hence the convolutional modules have been eliminated from the proposed model. The suggested model concludes with a multi-label classifier that utilizes a feed-forward network architecture. This classifier consists of two layers and employs a sigmoid activation function. The experimental findings provide evidence of the improved performance exhibited by the suggested model when compared to state-of-the-art approaches such as ResNet, VGG, DenseNet, and MobileNet, on the publicly available dataset ODIR-2019, and the proposed approach has outperformed the state-of-the-art algorithms in terms of Kappa, F1 score, AUC, and AVG.

Detecting Weak Physical Signal from Noise: A Machine-Learning Approach with Applications to Magnetic-Anomaly-Guided Navigation

Detecting a weak physical signal immersed in overwhelming noises entails separating the two, a task for which machine learning is naturally suited. In principle, such a signal is generated by a nonlinear dynamical system of intrinsically high dimension for which a mathematical model is not available, rendering unsuitable traditional linear or nonlinear state-estimation methods that require an accurate system model (e.g., extended Kalman filters). We exploit the architectures of reservoir computing and feed-forward neural networks (FNNs) with time-delayed inputs to solve the weak-signal-detection problem. As a prototypical example, we apply the machine-learning schemes to Earth's magnetic anomaly field-based navigation. In particular, the time series are collected from the interior of the cockpit of a flying aircraft during different maneuvering phases, where the overwhelmingly strong noise background is the result of other components of Earth's magnetic field and the fields generated by the electronic devices in the cockpit. We demonstrate that, when combined with the traditional Tolles-Lawson model for Earth's magnetic field, the articulated machine-learning schemes are effective for accurately detecting the weak anomaly field from the noisy time series. The schemes can be applied to detecting weak signals in other domains of science and engineering.

Spatial-Prior-Guided Attention for Small Object Detection in Overhead Catenary System

The small defects in overhead catenary system (OCS) can result in long time delays, economic loss and even passenger injury. However, OCS images exhibit great variations with complex background and oblique views which pose a great challenge for small defects detection in high-speed rail system. In this paper, we propose the spatial-prior-guided attention for small object detection in OCS with two main advantages: (1) The spatial-prior is proposed to retain the spatial information between small defects and the electric components in OCS. (2) Based on spatial-prior, the spatial-prior-guided attention model (SAM) is designed to highlight useful information in the features and suppress redundant features response. SAM can model the spatial relations progressively and can be integrated with state-of-the-art feed-forward network architecture with end-to-end training fashion. We conduct extensive experiments on both Split pin datasets and PASCAL–VOC datasets and achieve 97.2% and 79.5% mAP values, respectively. All the experiments demonstrate the competitive performance of our method.

Thermal management of nanoliquid forced convective flow over heated blocks in channel by using double elliptic porous objects

Effects of using nanoliquid and double porous elliptic objects on the flow separation and convective heat transfer control for flow over multiple heated blocks in channel are numerically investigated by using finite element method. Spherical, brick, blade and cylindrical shaped alumina particles are used in water up to solid volume fraction of 2%. Impacts of Reynolds number (50≤Re ≤ 500), permeability of the porous ellipses (10−5≤Da1≤5 × 10−2 and 10−5≤Da2≤5 × 10−2), aspect ratio of the ellipses (0.25≤AR1≤1.5, 0.25≤AR2≤1.5), distance between the ellipses (0.2H ≤ sx ≤ 1.6H) and distance of the first ellipse from inlet (−H ≤ mx ≤ H) on the hydro-thermal performance are explored. The highest performance improvement is obtained by using the cylindrical shaped particles in the base heat transfer fluid at the highest solid volume fraction and the amount of heat transfer enhancement are 33% and 40.5% for hot blocks while negligible pressure drop is observed. Varying the aspect ratio of the ellipses in the perpendicular direction to the flow, up to 25% and 30% heat transfer increment are obtained for hot block with slight rise of pressure coefficient. However, when varying the horizontal location of the porous ellipses in the channel and permeability of them, there is considerable rise of pressure drop. Variation of the average heat transfer from the hot blocks with varying permeability of the ellipses are obtained as 32% and 25%. Successful hydro-thermal performance estimation results are achieved by using artificial neural networks with feed-forward network architecture of 1 hidden layer and 17 neurons.

Dendritic normalisation improves learning in sparsely connected artificial neural networks.

Artificial neural networks, taking inspiration from biological neurons, have become an invaluable tool for machine learning applications. Recent studies have developed techniques to effectively tune the connectivity of sparsely-connected artificial neural networks, which have the potential to be more computationally efficient than their fully-connected counterparts and more closely resemble the architectures of biological systems. We here present a normalisation, based on the biophysical behaviour of neuronal dendrites receiving distributed synaptic inputs, that divides the weight of an artificial neuron's afferent contacts by their number. We apply this dendritic normalisation to various sparsely-connected feedforward network architectures, as well as simple recurrent and self-organised networks with spatially extended units. The learning performance is significantly increased, providing an improvement over other widely-used normalisations in sparse networks. The results are two-fold, being both a practical advance in machine learning and an insight into how the structure of neuronal dendritic arbours may contribute to computation.

Machine Learning Classifiers for Surface Crack Detection in Fracture Experiments

Correctly determining the onset of fracture is crucial when performing mechanical experiments. Commonly carried out by visual inspection, here an image-based machine learning approach is proposed to classify cracked and un-cracked specimens. It yields the potential to objectify and automate crack detection, thereby removing sources of uncertainty and error from the post-processing of experiments. More than 30’000 speckle-pattern images obtained from 77 experiments on three specimen geometries are evaluated. They comprise uniaxial tension, notched tension as well as axisymmetric V-bending experiments. Statistical texture features are extracted from all images. They include both first-order (variance, skewness, kurtosis) and higher-order statistical texture features, i.e. Haralick features. The discriminatory power of the texture information is evaluated based on the Fisher's Discriminant Ratio and feature correlations are identified and quantified. Image texture feature subsets of high discriminatory power are used to parse neural network architectures of different complexities from simple perceptron to feed-forward and cascade neural networks. It is found that a small subset of the investigated texture features is highly significant for all experiments. Using this feature subset in conjunction with multi-layer, non-linear and low complexity feed-forward network architectures classification accuracies in the order of 99% are obtained. At the same time, it is shown that linear classifiers are not sufficient to robustly distinguish the state of the specimens, even when high discriminatory power features are used.Graphical Abstract: [Display omitted]

NEUROEVOLUTION METHODS FOR FEEDFORWARD NEURAL NETWORKS

Various optimization methods and parameters of feedforward neural networks by evolutionary algorithms are considered. The analysis of neuroevolutionary methods, basic properties and applicability of various variants of evolutionary optimization with direct coding of chromosomes for various architectures of feedforward neural networks are presented. Operators of the genetic algorithm, advantages, disadvantages and features of their application are described for each method of neuroevolution. The comparison results are presented in tabular form.      

Neural variational sparse topic model for sparse explainable text representation

Texts are the major information carrier for internet users, from which learning the latent representations has important research and practical value. Neural topic models have been proposed and have great performance in extracting interpretable latent topics and representations of texts. However, there remain two major limitations: (1) these methods generally ignore the contextual information of texts and have limited feature representation ability due to the shallow feed-forward network architecture, (2) Sparsity of the representations in topic semantic space is ignored. To address these issues, in this paper, we propose a semantic reinforcement neural variational sparse topic model (SR-NSTM) towards explainable and sparse latent text representation learning. Compared with existing neural topic models, SR-NSTM models the generative process of texts with probabilistic distributions parameterized with neural networks and incorporates Bi-directional LSTM to embed contextual information at the document level. It achieves sparse posterior representations over documents and words with zero-mean Laplace distribution and topics with sparsemax. Moreover, we propose a supervised extension of SR-NSTM via adding the max-margin posterior regularization to tackle the supervised tasks. The neural variational inference method is utilized to learn our models efficiently. Experimental results on Web Snippets, 20Newsgroups, BBC, and Biomedical datasets demonstrate that the contextual information and revisiting generative process can improve the performance, leading to the competitive performance of our models in learning coherent topics and explainable sparse representations for texts.

Industrial, large-scale model predictive control with structured neural networks

The design of neural networks (NNs) is presented for treating large, linear model predictive control (MPC) applications that are out of reach with available quadratic programming (QP) solvers. First, we introduce a new feedforward network architecture that enables practitioners to obtain offset-free closed-loop performance with NNs. Second, we discuss the data generation procedure to sample the state space relevant to training the NNs based on anticipated online setpoint changes and plant disturbances. Third, we use the input-to-state stability results available in the MPC literature and establish robustness properties of NN controllers. Finally, we present illustrative simulation studies on process control examples. We apply the NN design approach and compare the performance with online QP based MPC on an industrial crude distillation unit model with 252 states, 32 control inputs, and a control-sample horizon length of 140. Parallel computing is used for data generation and graphical processing units are used for network training. Anticipated plant operational scenarios with setpoints and disturbances that may change during operation must be sampled for NN training. After the offline design phase, NNs execute MPC three to five orders of magnitude faster than an available QP solver with less than 1% loss in the closed-loop performance.

Pillar Number Plate Detection and Recognition in Unconstrained Scenarios

Overhead catenary system (OCS) images exhibit great variations with clutter backgrounds, complex scenes and oblique views which pose great difficulty for automatic pillar number plate (NP) detection and recognition (NPDAR). Although these tasks have an important and practical significance for railway transportation, little researches have been done on these fields. In this paper, we propose a complete automatic NPDAR system with two main advantages: (1) For detection task, we propose Skip Connection Attention Module (SCAM) for adaptive feature refinement. Based on SCAM, the Attention_Guided Feature Fusion (AFF) module is designed for building high-level feature maps at different scales. (2) A novel convolution module, width/height convolution module (W/H-CM) was designed for NP recognition to capture global feature information efficiently. The W/H-CM extracts contextual information from two other perspectives compared to common convolution operation and iteratively generates supplementary information, making the representation of features more comprehensive. Both of them can incorporate with state-of-art feed forward network architecture in an end-to-end training fashion. We conduct extensive experiments on both our datasets and standard benchmarks PASCAL-VOC, MS COCO to verify competitive performance of our method.

Deep Subpixel Mapping Based on Semantic Information Modulated Network for Urban Land Use Mapping

Mixed pixel problem is omnipresent in remote sensing images for urban land use interpretation due to the hardware limitations. Subpixel mapping (SPM) is a usual way to solve this problem by improving the observation scale and realizing a finer spatial resolution land cover mapping. Recently, deep learning-based subpixel mapping network (DLSMNet) was proposed, benefited from its strong representation and learning ability, to restore a visually pleasing finer mapping. However, the spatial context features of artifacts are usually aggregated and progressively lost during the forward pass of the network without sufficient representation, which make it difficult to be learned and restored. In this article, a semantic information modulated (SIM) deep subpixel mapping network (SIMNet) is proposed, which uses low-resolution semantic images as prior, to reinforce the representation of spatial context features. In SIMNet, SIM module is proposed to parametrically incorporate the semantic prior into the state-of-the-art (SOTA) feed forward network architecture in an end-to-end training fashion. Furthermore, stacked SIM module with residual blocks (SIM_ResBlock) is adopted to pass the representation of spatial context feature to the deep layers, to get it fully learned during backpropagation. Experiments have been implemented on three public urban scenario data sets, and the SIMNet generates a clearer outline of artificial facilities with sufficient spatial context, and is distinctive for even individual building, which is challenging for other SOTA DLSMNet. The results demonstrate that the proposed SIMNet is a promising way for high-resolution urban land use mapping from easily available lower resolution remote sensing images.Mixed pixel problem is omnipresent in remote sensing images for urban land use interpretation due to the hardware limitations. Subpixel mapping (SPM) is a usual way to solve this problem by improving the observation scale and realizing a finer spatial resolution land cover mapping. Recently, deep learning-based subpixel mapping network (DLSMNet) was proposed, benefited from its strong representation and learning ability, to restore a visually pleasing finer mapping. However, the spatial context features of artifacts are usually aggregated and progressively lost during the forward pass of the network without sufficient representation, which make it difficult to be learned and restored. In this article, a semantic information modulated (SIM) deep subpixel mapping network (SIMNet) is proposed, which uses low-resolution semantic images as prior, to reinforce the representation of spatial context features. In SIMNet, SIM module is proposed to parametrically incorporate the semantic prior into the state-of-the-art (SOTA) feed forward network architecture in an end-to-end training fashion. Furthermore, stacked SIM module with residual blocks (SIM_ResBlock) is adopted to pass the representation of spatial context feature to the deep layers, to get it fully learned during backpropagation. Experiments have been implemented on three public urban scenario data sets, and the SIMNet generates a clearer outline of artificial facilities with sufficient spatial context, and is distinctive for even individual building, which is challenging for other SOTA DLSMNet. The results demonstrate that the proposed SIMNet is a promising way for high-resolution urban land use mapping from easily available lower resolution remote sensing images.

Deep Local Volatility

Deep learning for option pricing has emerged as a novel methodology for fast computations with applications in calibration and computation of Greeks. However, many of these approaches do not enforce any no-arbitrage conditions, and the subsequent local volatility surface is never considered. In this article, we develop a deep learning approach for interpolation of European vanilla option prices which jointly yields the full surface of local volatilities. We demonstrate the modification of the loss function or the feed forward network architecture to enforce (hard constraints approach) or favor (soft constraints approach) the no-arbitrage conditions and we specify the experimental design parameters that are needed for adequate performance. A novel component is the use of the Dupire formula to enforce bounds on the local volatility associated with option prices, during the network fitting. Our methodology is benchmarked numerically on real datasets of DAX vanilla options.

Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization

Humans and most animals can learn new tasks without forgetting old ones. However, training artificial neural networks (ANNs) on new tasks typically causes them to forget previously learned tasks. This phenomenon is the result of "catastrophic forgetting," in which training an ANN disrupts connection weights that were important for solving previous tasks, degrading task performance. Several recent studies have proposed methods to stabilize connection weights of ANNs that are deemed most important for solving a task, which helps alleviate catastrophic forgetting. Here, drawing inspiration from algorithms that are believed to be implemented in vivo, we propose a complementary method: adding a context-dependent gating signal, such that only sparse, mostly nonoverlapping patterns of units are active for any one task. This method is easy to implement, requires little computational overhead, and allows ANNs to maintain high performance across large numbers of sequentially presented tasks, particularly when combined with weight stabilization. We show that this method works for both feedforward and recurrent network architectures, trained using either supervised or reinforcement-based learning. This suggests that using multiple, complementary methods, akin to what is believed to occur in the brain, can be a highly effective strategy to support continual learning.

Learning Context Flexible Attention Model for Long-Term Visual Place Recognition

Identifying regions of interest in an image has long been of great importance in a wide range of tasks, including place recognition. In this letter, we propose a novel attention mechanism with flexible context, which can be incorporated into existing feedforward network architecture to learn image representations for long-term place recognition. In particular, in order to focus on regions that contribute positively to place recognition, we introduce a multiscale context-flexible network to estimate the importance of each spatial region in the feature map. Our model is trained end-to-end for place recognition and can detect regions of interest of arbitrary shape. Extensive experiments have been conducted to verify the effectiveness of our approach and the results demonstrate that our model can achieve consistently better performance than the state of the art on standard benchmark datasets. Finally, we visualize the learned attention maps to generate insights into what attention the network has learned.

Desain Kontrol Sistem Eksitasi Steam Turbine Generator (Stg) Pada Combine Cycle Power Plant 120 Mw Pt.Krakatau Daya Istrik Menggunakan Jaringan Syaraf Tiruan Backpropagation

Excitation System is a vital part of generating electricity in the generator, the function of the excitation system in the generator is to control the output of the generator to remain stable at the changing system load. The excitation control system of the synchronous generator is one of the steps to improve the stability of the power system and to ensure the quality of the electricity generated by the network. In this research used one of the implementation of artificial intelligence technology is Artificial Neural Network designed as controller excitation system generator. The network is designed using a backpropagation learning algorithm, and the Lavenberg-Marquardt training algorithm with a feedforward multilayer network architecture. The result of simulated test and training using Matlab application obtained with presentation of system success equal to 96,387%.

Artificial Neural Network Model to Predict Reservoir Storage

The rapid growth in population increases water demand thus resulting in scarcity of water which is due to improper management rather than lack of resources. Reservoir is the most important source for surface water. So, reservoir storage plays a crucial role in efficient reservoir management. Artificial neural network (ANN) is capable of simulating reservoir storage capacity. So, in present work cascade forward back propagation and feed forward back propagation network artificial neural network (ANN) models with five different architecture in each by changing the number of hidden layer neurons were developed to predict Harangi reservoir storage, Karnataka, India. The first two years (2010–2012) data on daily basis was used for supervised training and remaining (2013–2014) data on daily basis was used in prediction. The predictive accuracy using the statistical parameters like correlation coefficient (R) and mean absolute percentage error (MAPE) were found within well acceptable limit. Results shows that, compared to the cascade forward back propagation network, feed forward back propagation network architecture of 6-5-1 ANN model with correlation coefficient (R) of 0.9882 and absolute mean percentage error (MAPE) of 5.225%. is best suited for present study.