Feed-forward Neural Network Algorithm Research Articles

Automatic defects detection in CFRP thermograms, using convolutional neural networks and transfer learning

Recent advancements in the field of Artificial Intelligence can support the post-processing of thermographic data, efficiently, especially for nonlinear or complex thermography scanning routines. This study proposes the implementation of an autonomous/intelligent post-processor that is capable of automatically detecting defects from given thermograms via a Convolutional Neural Networks (CNN) coding, in tandem with a Deep Feed Forward Neural Networks (DFF-NN) algorithm to estimate the defect depth as well. Thus, the proposed NN combination will process (detect and quantify) the defects from acquired thermograms in real-time, and without any human (inspector) intervention. The study shows that employing a pre-trained network, using a relatively small dataset of thermograms for training, can detect and quantify defects in thermographic sequences. In this paper, pre-trained networks with CIFAR-10 and ImageNet databases are used, and followed by a fine-tuning step of the later layers in the network; using a relatively small thermograms dataset. This text will also provide several in-depth studies to compare how transfer learning, state of the art object detection architectures, and the convolutional neural networks influence the performance of the trained post-processing system. The proposed post-processor applied to thermograms obtained from a pulsed-thermography setup testing a Carbon Fiber Reinforced Polymer (CFRP) sample with artificially created sub-surface defects validates the CNN approach.

Two noise tolerant incremental learning algorithms for single layer feed-forward neural networks

This paper focuses on noise resistant incremental learning algorithms for single layer feed-forward neural networks (SLFNNs). In a physical implementation of a well trained neural network, faults or noise are unavoidable. As biological neural networks have ability to tolerate noise, we would like to have a trained neural network that has certain ability to tolerate noise too. This paper first develops a noise tolerant objective function that can handle multiplicative weight noise. We assume that multiplicative weight noise exist in the weights between the input layer and the hidden layer, and in the weights between the hidden layer and the output layer. Based on the developed objective function, we propose two noise tolerant incremental extreme learning machine algorithms, namely weight deviation incremental extreme learning machine (WDT-IELM) and weight deviation convex incremental extreme learning machine (WDTC-IELM). Compared to the original extreme learning machine algorithms, the two proposed algorithms have much better ability to tolerate the multiplicative weight noise. Several simulations are carried out to demonstrate the superiority of the two proposed algorithms.

Transformed Secure Feed Forward Supervised Learning Method for Authentication in Multi-Model Biometric System

Biometric is an automated detection of the characteristics of an individual on the basis of the biological and social features. Detection of the uni-modal biometric system is based on the biometric data of an individual. Some issue of distortion level spoofing threats are more accessible to biometric data. Some of the issues overcome by multimodal biometric scheme in which signature of the biometric data are determine for better security of the data. Multimodal biometric is used on variety of the application areas which are human computer interface, detection of the sensor through unique method. The physical and social characteristics are used for the identification of an individual using multimodal biometric system. Multi-model biometric system applications are security system developed in banking sectors, business phase and Industry (MNC) companies. In existing work, using ESVM method to recognize the biometric traits and problem occurs in existing phase is distortion and degrades the image quality present and reduces the recognition rate and high error rates. In proposed research, determined the biometric features finger print, face and iris through CASIA dataset. Then, distortion rate is recognised through salt and pepper method and removal of interference using filtration technique. After that, discrete wavelet transformation is used for the extraction of the features of the biometric system through face, fingerprint and eye that determine the graphical features. Along with that, feed forward neural network algorithm developed for classification and recognition of multi modal biometric behaviour characteristics. The Encrypted NN method conducts simulation work on the metrics like as a recognition rate, true positive rate and computation time. The experimental results demonstrate that Encrypted NN method is able to enhance the image quality, recognition rate and TPR and reduces the computational time of Multi-model Biometric System when compared with existing work and simulation tool used MATLAB 2016a.

A Heterogeneous AdaBoost Ensemble Based Extreme Learning Machines for Imbalanced Data

Extreme learning machine (ELM) is an effective learning algorithm for the single hidden layer feed-forward neural network (SLFN). It is diversified in the form of kernels or feature mapping functions, while achieving a good learning performance. It is agile in learning and often has good performance, including kernel ELM and Regularized ELM. Dealing with imbalanced data has been a long-term focus for the learning algorithms to achieve satisfactory analytical results. It is obvious that the unbalanced class distribution imposes very challenging obstacles to implement learning tasks in real-world applications, including online visual tracking and image quality assessment. This article addresses this issue through advanced diverse AdaBoost based ELM ensemble (AELME) for imbalanced binary and multiclass data classification. This article aims to improve classification accuracy of the imbalanced data. In the proposed method, the ensemble is developed while splitting the trained data into corresponding subsets. And different algorithms of enhanced ELM, including regularized ELM and kernel ELM, are used as base learners, so that an active learner is constructed from a group of relatively weak base learners. Furthermore, AELME is implemented by training a randomly selected ELM classifier on a subset, chosen by random re-sampling. Then, the labels of unseen data could be predicted using the weighting approach. AELME is validated through classification on real-world benchmark datasets.

Abnormal Behavior Recognition Based on Transfer Learning

In order to identify abnormal behavior in image samples, a transfer learning recognition model based on Inception-V3 neural network is proposed in this paper. The whole process consists of two parts: 1) Inception-V3 neural network is used to extract the features of image samples.2) The abnormal behavior recognition model is established to classify the obtained features. In this paper, three-layer feedforward neural network algorithm is used to compare with other common machine learning classification models (k-nearest neighbor algorithm, random forest, LightGBM, SVM, two-layer feedforward neural network).The results show that the performance of anomalous behavior recognition based on three-layer feedforward neural network is better than other classification models.

Artificial Neural Networks Approach for a Multi-Objective Cavitation Optimization Design in a Double-Suction Centrifugal Pump

Double-suction centrifugal pumps are widely used in industrial and agricultural applications since their flow rate is twice that of single-suction pumps with the same impeller diameter. They usually run for longer, which makes them susceptible to cavitation, putting the downstream components at risk. A fast approach to predicting the Net Positive Suction Head required was applied to perform a multi-objective optimization on the double-suction centrifugal pump. An L32 (84) orthogonal array was designed to evaluate 8 geometrical parameters at 4 levels each. A two-layer feedforward neural network and genetic algorithm was applied to solve the multi-objective problem into pareto solutions. The results were validated by numerical simulation and compared to the original design. The suction performance was improved by 7.26%, 3.9%, 4.5% and 3.8% at flow conditions 0.6Qd, 0.8Qd, 1.0Qd and 1.2Qd respectively. The efficiency increased by 1.53% 1.0Qd and 1.1% at 0.8Qd. The streamline on the blade surface was improved and the vapor volume fraction of the optimized impeller was much smaller than that of the original impeller. This study established a fast approach to cavitation optimization and a parametric database for both hub and shroud blade angles for double suction centrifugal pump optimization design.

Incremental Multiple Hidden Layers Regularized Extreme Learning Machine Based on Forced Positive-Definite Cholesky Factorization

The theory and implementation of extreme learning machine (ELM) prove that it is a simple, efficient, and accurate machine learning method. Compared with other single hidden layer feedforward neural network algorithms, ELM is characterized by simpler parameter selection rules, faster convergence speed, and less human intervention. The multiple hidden layer regularized extreme learning machine (MRELM) inherits these advantages of ELM and has higher prediction accuracy. In the MRELM model, the number of hidden layers is randomly initiated and fixed, and there is no iterative tuning process. However, the optimal number of hidden layers is the key factor to determine the generalization ability of MRELM. Given this situation, it is obviously unreasonable to determine this number by trial and random initialization. In this paper, an incremental MRELM training algorithm (FC-IMRELM) based on forced positive-definite Cholesky factorization is put forward to solve the network structure design problem of MRELM. First, an MRELM-based prediction model with one hidden layer is constructed, and then a new hidden layer is added to the prediction model in each training step until the generalization performance of the prediction model reaches its peak value. Thus, the optimal network structure of the prediction model is determined. In the training procedure, forced positive-definite Cholesky factorization is used to calculate the output weights of MRELM, which avoids the calculation of the inverse matrix and Moore-Penrose generalized inverse of matrix involved in the training process of hidden layer parameters. Therefore, FC-IMRELM prediction model can effectively reduce the computational cost brought by the process of increasing the number of hidden layers. Experiments on classification and regression problems indicate that the algorithm can be effectively used to determine the optimal network structure of MRELM, and the prediction model training by the algorithm has excellent performance in prediction accuracy and computational cost.

Generalized Dai-Yuan conjugate gradient algorithm for training multi-layer feed-forward neural networks

In this paper, we will present different type of CG algorithms depending on Peary conjugacy condition. The new conjugate gradient training (GDY) algorithm using to train MFNNs and prove it's descent property and global convergence for it and then we tested the behavior of this algorithm in the training of artificial neural networks and compared it with known algorithms in this field through two types of issues.

Wavelet energy moment and neural networks based particle swarm optimisation for transmission line protection

In this study, a combined approach of discrete wavelet transform analysis and a feed forward neural networks algorithm to detect and classify transmission line faults. The proposed algorithm uses a multi -resolution analysis decoposition of three-phasecurrents only to calculate the wavelet energy moment of detailed coefficients. In comparison with the energy spectrum, the energy moment could reveal the energy distribution features better, which is beneficial when extracting signal features. Theapproach use particle swarm optimization algorithm to train a feed forward neural network. The goal is the enhancement of the convergence rate, learning process and fill up the gap of local minimum point.The purposed scheme consists of two FNNs, one for detecting and another for classifying all the ten types of faults using Matlab/Simulink. The proposed algorithm have been extensively tested on a system 400 kV, 3 phases, 100 km line consideringvarious fault parameter variations.

An unsupervised parameter learning model for RVFL neural network

With the direct input–output connections, a random vector functional link (RVFL) network is a simple and effective learning algorithm for single-hidden layer feedforward neural networks (SLFNs). RVFL is a universal approximator for continuous functions on compact sets with fast learning property. Owing to its simplicity and effectiveness, RVFL has attracted significant interest in numerous real-world applications. In reality, the performance of RVFL is often challenged by randomly assigned network parameters. In this paper, we propose a novel unsupervised network parameter learning method for RVFL, named sparse pre-trained random vector functional link (SP-RVFL for short) network. The proposed SP-RVFL uses a sparse autoencoder with ℓ1-norm regularization to adaptively learn superior network parameters for specific learning tasks. By doing so, the learned network parameters in SP-RVFL are embedded with the valuable information of input data, which alleviate the randomly generated parameter issue and improve the algorithmic performance. Experiments and comparisons on 16 diverse benchmarks from different domains confirm the effectiveness of the proposed SP-RVFL. The corresponding results also demonstrate that RVFL outperforms extreme learning machine (ELM).

A Biological Mechanism Based Structure Self-Adaptive Algorithm for Feedforward Neural Network and Its Engineering Applications

Feedforward neural network (FNN) is an information processing system that simulates human brain function to a certain extent by referring to the structure of a biological neural network and the working mechanism of biological neurons. As the most important part, the architecture of FNN essentially influences its application performance. This paper proposed a structure self-adaptive algorithm for FNN (SSAFNN) based on biological mechanism. First, self-adaptive neuron growing and pruning indexes are proposed based on the idea of biological neuron grow factor and neuron competition, respectively. The FNN structure is dynamically adjusted according to the growing and pruning indexes of hidden neurons. Second, the connect weights of FNN are automatically adjusted during the self-organizing process and trained by gradient descent method during the learning process. Third, the theoretical analysis is given that this proposed algorithm not only optimized the network structure but also ensured the convergence and performance of FNN. The proposed SSAFNN is tested on the benchmark problems in the field of classification and prediction and is applied in the engineering problems of anchor bolt non-destructive testing and wastewater effluent ammonia nitrogen predicting. The experiment results reveal the good performance and potentiality of SSAFNN in industrial applications.

Machine Learning Algorithms for Short-Term Load Forecast in Residential Buildings Using Smart Meters, Sensors and Big Data Solutions

In this paper, we propose a scalable Big Data framework that collects the data from smart meters and weather sensors, pre-processes and loads it into a NoSQL database that is capable to store and further process large volumes of heterogeneous data. Then, a set of Machine Learning (ML) algorithms are designed and implemented to determine the load profiles and forecast the electricity consumption for residential buildings for the next 24 hours. For the Short-Term Load Forecast (STLF), a Feed-Forward Artificial Neural Network (FF-ANN) algorithm with backtracking adjustment of the learning rate that extends and optimizes the Nesterov learning method is proposed. Its performance is compared with six algorithms, i.e. FF-ANN with well-known learning methods, namely Momentum and Nesterov, Non-linear AutoRegressive with eXogenous (NARX), Deep Neural Network (DNN), Gradient Tree Boosting (GTB) and Random Forests (RF) that are competitive and powerful ML algorithms which have been successfully used for load forecast. Hence, for STLF, the seven algorithms are executed simultaneously and the best one is automatically selected considering its accuracy in terms of Root Mean Square Errors (RMSE). The proposed methodology contains the steps required to implement the Big Data framework, i.e. data pre-processing, transformation and loading, the configuration of the ML algorithms for dimensionality reduction, clustering, STLF with different algorithms from which the Best Performant Algorithm (BPA) is automatically selected to provide STLF for the next 24 hours. The methodology is ultimately tested considering a real case of a residential smart building.

NOVEL PRE-PROCESSING FRAMEWORK TO IMPROVE CLASSIFICATION ACCURACY IN OPINION MINING

The growth of information technology led to the Internet development that in turn helped people in many ways. The major one is to express their views about the products and services through reviews, blogs, feedback, and comments on the website and in social media. The buyers are forced to go through investigation on these reviews/blogs, before choosing any product or service. Out of all online services, Mobile learning app places a vital role to increase the thirst for knowledge. But to identify the suitable mobile learning app, the opinions of the existing customers need to be mined. This research paper analyzes the mobile learning reviews which are available in the corpus. A novel preprocessing framework is proposed in this paper to improve classification accuracy in the dataset - mobile learning app review dataset. The corpus dimension is reduced using SVD through which, the data is prepared for mining. The classification accuracy is evaluated by applying Multinomial Naïve Bayes, Random Forest data mining algorithms and Learning Vector Quantization (LVQ), Elman Neural Network (ENN), Feed Forward Neural Network (FFNN) algorithms with the dataset obtained by the proposed processing method.

An Indoor and Outdoor Positioning Using a Hybrid of Support Vector Machine and Deep Neural Network Algorithms

Indoor and outdoor positioning lets to offer universal location services in industry and academia. Wi-Fi and Global Positioning System (GPS) are the promising technologies for indoor and outdoor positioning, respectively. However, Wi-Fi-based positioning is less accurate due to the vigorous changes of environments and shadowing effects. GPS-based positioning is also characterized by much cost, highly susceptible to the physical layouts of equipment, power-hungry, and sensitive to occlusion. In this paper, we propose a hybrid of support vector machine (SVM) and deep neural network (DNN) to develop scalable and accurate positioning in Wi-Fi-based indoor and outdoor environments. In the positioning processes, we primarily construct real datasets from indoor and outdoor Wi-Fi-based environments. Secondly, we apply linear discriminate analysis (LDA) to construct a projected vector that uses to reduce features without affecting information contents. Thirdly, we construct a model for positioning through the integration of SVM and DNN. Fourthly, we use online datasets from unknown locations and check the missed radio signal strength (RSS) values using the feed-forward neural network (FFNN) algorithm to fill the missed values. Fifthly, we project the online data through an LDA-based projected vector. Finally, we test the positioning accuracies and scalabilities of a model created from a hybrid of SVM and DNN. The whole processes are implemented using Python 3.6 programming language in the TensorFlow framework. The proposed method provides accurate and scalable positioning services in different scenarios. The results also show that our proposed approach can provide scalable positioning, and 100% of the estimation accuracies are with errors less than 1 m and 1.9 m for indoor and outdoor positioning, respectively.

Financial prediction using back propagation neural networks with opposition based learning

Stock price prediction has recently brought together significant attention among the researchers. In the past decades, the stock price prediction has contributed a primary role in the stock market. The shareholder and investigators should give concentration a reliable method to predict stock price/index. An accurate prediction is an important to enlarge an effective market trading strategies. A back propagation neural network is widely used well known multi-layer supervised feed forward neural network algorithm since its simplicity and high problem-solving ability. In the traditional back propagation neural network, weight updating done by gradient decent based learning algorithm which is falling into local minima and learning rate is slow. Hence, keep away from above mentioned drawbacks; the opposition based learning (OBL) algorithm is used for weight adjustment in a back propagation neural network. The empirical result shows that the proposed prediction model demonstrates a superior performance in financial time series forecasting. For evaluating the performance of the proposed model, the empirical research is applied to well known stock market data sets such as S & P BSE Sensex and Nifty 50.

Comparison of Response Surface Methodology (RSM) and Artificial Neural Network (ANN) in modelling of waste coconut oil ethyl esters production

ABSTRACTThe attributes of renewability and environmental friendliness have made ethanol a preferable alternative to methanol in the production of biodiesel from lipid feedstocks. For the first time, this study adopted Response Surface Methodology (RSM) and Artificial Neural Network (ANN) to model coconut oil ethyl ester (CNOEE) yield. Transesterification parameters such as reaction temperature and ethanol/coconut oil molar ratio and catalyst dosage were varied. Maximum CNOEE yield of 96.70% was attained at 73 °C reaction temperature, 11.9:1 molar ratio, and catalyst dosage of 1.25 wt. %. The experimental yield was in agreement with the predicted yield. Central Composite Design was adopted to develop the RSM while feed-forward back propagation neural network algorithm was employed for the ANN model. Statistical indices were employed to compare the models. The computed coefficient of determination (R2) of 0.9564, root-mean-squarce-error (RMSE) of 0.72739, standard error of prediction (SEP) of 0.008021, mean average error (MAE) of 0.612, and average absolute deviation (AAD) of 0.674901 for RSM model compared to those of R2 (0.9980), RMSE (0.68615), SEP (0.007567), MAE (0.325), and AAD (0.3877) for ANN indicated the superiority of the ANN model over the RSM model. The key fuel properties of the CNOEE met with those of biodiesel international standards.

An effective hierarchical extreme learning machine based multimodal fusion framework

Abstract Deep learning has been successfully applied to multimodal representation learning. Similar with single modal deep learning method, such multimodal deep learning methods consist of a greedy layer-wise feedforward propagation and a backpropagation (BP) fine-tune conducted by diverse targets. These models have the drawback of time consuming. While, extreme learning machine (ELM) is a fast learning algorithm for single hidden layer feedforward neural network. And previous works has shown the effectiveness of ELM based hierarchical framework for multilayer perceptron. In this paper, we introduce an ELM based hierarchical framework for multimodal data. The proposed architecture consists of three main components: (1) self-taught feature extraction for specific modality by an ELM-based sparse autoencoder, (2) fused representation learning based on the features learned by previous step and (3) supervised feature classification based on the fused representation. This is an exact feedforward framework that once a layer is established, its weights are fixed without fine-tuning. Therefore, it has much better learning efficiency than the gradient based multimodal deep learning methods. We conduct experiments on MNIST, XRMB and NUS datasets, the proposed algorithm obtains faster convergence and achieves better classification performance compared with the other existing multimodal deep learning models.

Sensitivity analysis of parameters affecting scour depth around bridge piers based on the non-tuned, rapid extreme learning machine method

The extreme learning machine (ELM) is a new, non-tuned and fast training algorithm for feedforward neural networks (FFNN). It is highly precise and randomly produces the input weights of single-layer FFNN. In the current study, the scour depth around bridge piers is predicted by ELM as a powerful method of nonlinear system modeling. To predict scour depth, the effective dimensionless parameters are determined through dimensional analysis. Due to the complexity of scour mechanisms around bridges, different models with diverse input numbers are presented. In 5 categories, 31 different models were obtained for modeling and ELM analysis. Following the training and validation of each model presented, the optimum model was selected from each of the 5 categories and its relationship to the respective category was identified to help determine scour depth in practical engineering. For the best models presented in the different input modes, new explicit expressions were deduced. The results show that the most important parameters affecting relative scour depth (ds/y) include ratio of pier width to flow depth (D/y) and ratio of pier length to flow depth (L/y) (RMSE = 0.08; MARE = 0.0.35). The ELM performance was compared for a range of pier geometries with regression-based equations. The results confirm that ELM outperforms other methods.

Feed-forward neural network training using sparse representation

The feed-forward neural network (FNN) has drawn great interest in many applications due to its universal approximation capability. In this paper, a novel algorithm for training FNNs is proposed using the concept of sparse representation. The major advantage of the proposed algorithm is that it is capable of training the initial network and optimizing the network structure simultaneously. The proposed algorithm consists of two core stages: structure optimization and weight update. In the structure optimization stage, the sparse representation technique is employed to select important hidden neurons that minimize the residual output error. In the weight update stage, a dictionary learning based method is implemented to update network weights by maximizing the output diversity from hidden neurons. This weight-updating process is designed to improve the performance of the structure optimization. Based on several benchmark classification and regression problems, we present experimental results comparing the proposed algorithm with state-of-the-art methods. Simulation results show that the proposed algorithm offers comparative performance in terms of the final network size and generalization ability.

Machine learning applied to broadband sound propagation on the New England Shelf

Addressed is to what degree can variability of an ocean waveguide, such as that associated with the seabed, be inferred by estimating a mapping y = f(x,θ) for input/category pairs (x,y) and parameterization θ. Feedforward neural network algorithms are employed to learn the optimal parameterization θ approximating f. The mapping includes a chain of functions with hidden layers or functions not directly related to the input. Supervised learning is applied to broadband propagation in a shallow water environment called the New England Mudpatch whose acoustic characterization has been studied continuously since 2015. In 2017 about 400 MK-64 Signal Underwater Sound (SUS) and Combustive Sound Sources (CSS) were deployed in a 30 × 11 km2 area with horizontal variability of the seabed for the purpose of inferring the statistics of the seabed characterization from the observed statistics of the acoustic field. The goal is to connect θ to physical source and waveguide parameter values. [Work supported by ONR.]