Hidden Output Layer Research Articles

Back propagation artificial neural network (BP-ANN) for prediction of the quality of gamma-irradiated smoked bacon

This study investigated the effect of gamma irradiation on smoked bacon quality during storage and developed a multi-quality prediction model based on gamma irradiation. Gamma irradiation reduced moisture content and improved the microbial safety of smoked bacon. It also accelerated protein and lipid oxidation and altered free amino acids and fatty acids composition. It was effective in slowing down quality deterioration and sensory quality decline during storage. The backpropagation artificial neural network (BP-ANN) model was constructed by using physical and chemical indicators, irradiation dose, and storage time as input variables, and the total number of colonies and sensory scores as output layers. The transfer functions of the input-hidden layer and hidden-output layer were ReLu and Sigmoid, respectively. There were 13 neurons in the hidden layer. Results showed that BP-ANN based on physical and chemical indicators, irradiation dose, and storage time had great potential in predicting the multiple quality of smoked bacon.

Steam blanching strengthened far-infrared drying of broccoli: Effects on drying kinetics, microstructure, moisture migration, and quality attributes

The effects of steam blanching (SB) at various times (0, 30, 60, 90, 120, and 150 s) on the far-infrared drying kinetics, quality properties of broccoli, and relevant mechanisms were investigated. Compared with the untreated samples, SB increased the drying rate, and reduced the drying time by 9.26%-48.15%. The back propagation-artificial neural networks (BP-ANN) was optimized and performed the good fitting performance between drying time, blanching time and moisture ratio, with the best training function of trainlm, transfer function of tangsig in input-hidden layer and purelin in hidden-output layer, and 10 hidden layer neurons. Meanwhile, SB destroyed the cellular structure, that caused the collapse of the tissue and the increase of water freedom, and which were intensified by prolonging the blanching time. Blanched sample after drying had better color, higher chlorophyll and vitamin C contents, while the retention of sulforaphane decreased slightly after blanching compared with the untreated broccoli.

A novel correlation Gaussian process regression-based extreme learning machine.

An obvious defect of extreme learning machine (ELM) is that its prediction performance is sensitive to the random initialization of input-layer weights and hidden-layer biases. To make ELM insensitive to random initialization, GPRELM adopts the simple an effective strategy of integrating Gaussian process regression into ELM. However, there is a serious overfitting problem in kernel-based GPRELM (kGPRELM). In this paper, we investigate the theoretical reasons for the overfitting of kGPRELM and further propose a correlation-based GPRELM (cGPRELM), which uses a correlation coefficient to measure the similarity between two different hidden-layer output vectors. cGPRELM reduces the likelihood that the covariance matrix becomes an identity matrix when the number of hidden-layer nodes is increased, effectively controlling overfitting. Furthermore, cGPRELM works well for improper initialization intervals where ELM and kGPRELM fail to provide good predictions. The experimental results on real classification and regression data sets demonstrate the feasibility and superiority of cGPRELM, as it not only achieves better generalization performance but also has a lower computational complexity.

Structural Damage Detection Based on One-Dimensional Convolutional Neural Network

This paper proposes a structural damage detection method based on one-dimensional convolutional neural network (CNN). The method can automatically extract features from data to detect structural damage. First, a three-layer framework model was designed. Second, the displacement data of each node was collected under the environmental excitation. Then, the data was transformed into the interlayer displacement to form a damage dataset. Third, in order to verify the feasibility of the proposed method, the damage datasets were divided into three categories: single damage dataset, multiple damage dataset, and damage degree dataset. The three types of damage dataset can be classified by the convolutional neural network. The results showed that the recognition accuracy is above 0.9274. Thereafter, a visualization tool called “t-SNE” was employed to visualize the raw data and the output data of the convolutional neural network. The results showed that the feature extraction ability of CNN is excellent. However, there are many hidden layers in a CNN. The outputs of these hidden layers are invisible. In the last section, the outputs of hidden layers are visualized to understand how the convolutional neural networks work.

Prediction of Direct Tensile Strength of Bamboo Fiber Reinforced Concrete Using Artificial Neural Network

Bamboo is among the most economically-significant non-timber forest products in the Philippines, and its fiber can be used for concrete reinforcement, known as bamboo fiber reinforced concrete (BFRC). However, BFRC needs further exploration, and its direct tensile strength is an essential factor that needs to be studied. The present study assessed the direct tensile strength of BFRC utilizing finite element modeling (FEM) and Artificial Neural Network (ANN). A positive correlation was found between compressive and direct tensile strength of the BFRC. The optimum ANN model was obtained from the trial-error approach. This model has 5-5-1 (input-hidden neurons-output) network topology and uses the tan sigmoid transfer function for both input-hidden and hidden-output layer which brought the most accurate prediction. The Levenberg–Marquardt (LM) provided the best optimization performance among other algorithms. The best model has a mean squared error (MSE) of 0.0000287 and r values of 0.98619, 0.99936, 0.99516, and 0.98565 for training, testing, validating, and overall. Sensitivity Analysis shows that compressive can significantly change the system’s performance even with a small change in its parameter value.

Extreme Learning With Metaheuristic Optimization for Exchange Rate Forecasting

Model with better learning ability and lower structural complexity is desirous for accurate exchange rate forecasting. Faster convergence to optimal solutions has always been a goal for the researcher in building forecasting models. And this is achieved by extreme learning machines (ELMs) due to their single hidden layer architecture and superior generalization ability. ELM is a simple training algorithm used to find the hidden-output layer weights by a random selection of input-hidden layer weights. Metaheuristics algorithms like Fireworks algorithm (FWA), Chemical reaction optimization (CRO), and Teaching learning-based optimization (TLBO) are employed to pre-train the ELM owing to their fewer optimizing parameters. This article aims to pre-train ELM using the said metaheuristics separately, ensuring the optimal solution of a single feedforward network (SLFN) with improved accuracy. The pre-trained ELMs provide accurate results. The same was verified using other primitive optimization algorithms

An Online Data-Driven LPV Modeling Method for Turbo-Shaft Engines

The linear parameter-varying (LPV) model is widely used in aero engine control system design. The conventional local modeling method is inaccurate and inefficient in the full flying envelope. Hence, a novel online data-driven LPV modeling method based on the online sequential extreme learning machine (OS-ELM) with an additional multiplying layer (MLOS-ELM) was proposed. An extra multiplying layer was inserted between the hidden layer and the output layer, where the hidden layer outputs were multiplied by the input variables and state variables of the LPV model. Additionally, the input layer was set to the LPV model’s scheduling parameter. With the multiplying layer added, the state space equation matrices of the LPV model could be easily calculated using online gathered data. Simulation results showed that the outputs of the MLOS-ELM matched that of the component level model of a turbo-shaft engine precisely. The maximum approximation error was less than 0.18%. The predictive outputs of the proposed online data-driven LPV model after five samples also matched that of the component level model well, and the maximum predictive error within a large flight envelope was less than 1.1% with measurement noise considered. Thus, the efficiency and accuracy of the proposed method were validated.

Creating synthetic minority class samples based on autoencoder extreme learning machine

This paper reports a new method (simplified as AE-ELM-SynMin) to create the Synthetic Minority class samples for imbalanced classification based on AutoEncoder Extreme Learning Machine (AE-ELM). AE-ELM-SynMin first trains an AE-ELM which is a special ELM with the same input and output, i.e., the original minority class samples. Second, the crossover, mutation and filtration operations are conducted on the hidden-layer output of AE-ELM and then the synthetic hidden-layer output is obtained. Third, the synthetic minority class samples are created by decoding the synthetic hidden-layer output with output-layer weights of AE-ELM. AE-ELM-SynMin guarantees that the synthetic minority class has the higher information amount than original minority class and meanwhile keeps the consistent probability distribution with the original minority class. The experimental results demonstrate the better imbalanced classification performances of AE-ELM-SynMin in comparison with the regular synthetic minority over-sampling technique (Regular-SMOTE) and its variants, e.g., Borderline-SMOTE, Random-SMOTE, and SMOTE-IPF.

Research on HR management based on RNN expectation theory

RNN convolutional neural network is an important part of deep learning algorithm. Compared with fully connected neural network, the biggest difference is that the hidden units are not independent of each other. Not only the hidden layer neurons are irrelated to each other, but also the state of the current hidden layer cells is affected by the historical input data. This feature enables it to extract the temporal relationship of temporal data structure. One of the current outputs of one sequence is also related to the previous output. The specific expression is that the network will memorize the information before and apply it to the calculation of current output, the nodes between hidden layers are no longer connected but connected. Moreover, the input of hidden layer includes not only the output of input layer but also the output of hidden layer at the last time. RNN neural network is widely used in natural language processing (NLP): machine translation, speech recognition, image description generation, text similarity calculation. The DAFDC-RNN model in this paper is verified by the financial and real estate industry data from 2019 to 2020, which shows that the model has excellent stability and sensitivity. Compared with CNN model, it increases by 34.3% and 24.9%. The advantages of data processing as follow: it can process a large number of time series data, the processing capacity of data set is large, it can process 1000 + data, and it has more data base for time series prediction. The disadvantage as follow: Compared with CNN model, the working time of CNN model is increased by 48%, but the model has strong stability and adaptability. Taking the data from 2019 to 2020 as the input of DAFDC-RNN time series forecasting model, the salary forecast of real estate and financial industry in 2021 is obtained. Based on this data, the industry expectation of Finance and real estate industry under human resource management under expectation theory is reduced.

Uncertainty of Rules Extracted from Artificial Neural Networks

ABSTRACT Artificial neural networks evolve into deep learning recently and perform well in various fields, such as image and speech recognition and translation. However, there is a problem that it is difficult for a person to understand what exactly the trained knowledge of an artificial neural network. As one of the methods for solving the problem of the artificial neural network, rule extraction methods had been devised. In this study, rules are extracted from artificial neural networks using ordered-attribute search (OAS) algorithm, which is one of the methods of extracting rules from trained neural networks, and the rules are analyzed to improve the extracted rules. As a result, we found that when the output value of the hidden layer has an intermediate value that is not close to 0 or 1 after passing through the sigmoid function, the problem of rule uncertainty occurs and affects the accuracy of the rules. In order to solve the uncertainty problem of the rules, we applied the hidden unit clarification method and suggested that it is possible to extract the efficient rule by binarizing the hidden layer output value. In addition, we extracted CDRPs (critical data routing paths) from the trained neural networks and used CDRPs to prune the extracted rules, which showed that the uncertainty problem of rules can be improved.

A new method for measuring the distribution consistency of mixed-attribute data sets

The measurement of data distribution consistency is a key problem in the process of generating random sample partition (RSP) of big data. How to measure the distribution consistency of mixed-attribute data sets reasonably and effectively is the focus of current research on RSP technology. This paper proposes a new method to measure the distribution consistency of mixed-attribute data sets based on deep encoding and maximum mean discrepancy (DE-MMD). Firstly, we conduct the one-hot encoding to transform the original data set with discrete attributes into the one-hot encoding data set. Then, we construct and train an autoencoder with single hidden layer based on the one-hot encoding data and thus we get the corresponding deep encoding data set by representing the original data set with hidden layer output. Finally, we measure the distribution consistency of mixed-attribute data sets based on the corresponding deep encoding data sets by using the maximum mean discrepancy index. On 4 benchmark mixed-attribute data sets, which are Adult、 Australian、 CRX and German, we compare the measure performances of DE-MMD method with those of the one-hot encoding-based MMD (OE-MMD) method and the binarization-based similarity measure (BSM) method. The experimental results show that the proposed method can measure the distribution consistency of mixed-attribute data sets more accurately and more effectively than OE-MMD and BSM methods.

Bayesian Learning of LF-MMI Trained Time Delay Neural Networks for Speech Recognition

Discriminative training techniques define state-of-the-art performance for automatic speech recognition systems. However, they are inherently prone to overfitting, leading to poor generalization performance when using limited training data. In order to address this issue, this paper presents a full Bayesian framework to account for model uncertainty in sequence discriminative training of factored TDNN acoustic models. Several Bayesian learning based TDNN variant systems are proposed to model the uncertainty over weight parameters and choices of hidden activation functions, or the hidden layer outputs. Efficient variational inference approaches using as few as one single parameter sample ensure their computational cost in both training and evaluation time comparable to that of the baseline TDNN systems. Statistically significant word error rate (WER) reductions of 0.4%-1.8% absolute (5%-11% relative) were obtained over a state-of-the-art 900 h speed perturbed Switchboard corpus trained baseline LF-MMI factored TDNN system using multiple regularization methods including F-smoothing, L2 norm penalty, natural gradient, model averaging and dropout, in addition to i-Vector plus learning hidden unit contribution (LHUC) based speaker adaptation and RNNLM rescoring. The efficacy of the proposed Bayesian techniques is further demonstrated in a comparison against the state-of-the-art performance obtained on the same task using the most recent hybrid and end-to-end systems reported in the literature. Consistent performance improvements were also obtained on a 450-h HKUST conversational Mandarin telephone speech recognition task. On a third cross domain adaptation task requiring rapidly porting a 1000-h LibriSpeech data trained system to a small DementiaBank elderly speech corpus, the proposed Bayesian TDNN LF-MMI systems outperformed the baseline system using direct weight fine-tuning by up to 2.5% absolute WER reduction.

Uncertain data stream algorithm based on clustering RBF neural network

In this paper, the existing algorithms for modeling uncertain data streams based on radial basis function neural networks have problems of low accuracy, weak stability and slow speed. A new clustering method for uncertain data streams is proposed. Radial basis function neural network of the algorithm. The algorithm firstly models the uncertain data stream, then combines the fuzzy theory and the neural network principle to obtain the radial basis function neural network, and then obtains the radial basis function neural network through the clustering algorithm of the regular tetrahedral uncertain vector. The central weight and width weights ultimately result in hidden layer output and output layer output results. The experimental results show that the proposed algorithm is an effective algorithm for modeling uncertain data streams using clustering radial basis function neural networks. It has higher precision, stability and speed than similar algorithms.

Application of artificial neural networks to predict multiple quality of dry-cured ham based on protein degradation

This study investigated protein degradation and quality changes during the processing of dry-cured ham, and then established the multiple quality prediction model based on protein degradation. From the raw material to the curing period, proteolysis index of external samples were higher than that of internal samples, however, the difference gradually decreased from the drying period to the maturing period. Protein degradation can be used as indicators for controlling quality of the hams. With protein degradation index as input variables, the back propagation-artificial neural networks (BP-ANN) models were optimized, with training function of trainlm, transfer function of logsig in input-hidden layer and tansig in hidden-output layer, and 20 hidden layer neurons. Furthermore, the relative errors of predictive data and experimental data of 12 samples were approximately 0 with the BP-ANN model. Results indicated that the BP-ANN has great potential in predicting multiple quality of dry-cured ham based on protein degradation.

Two-Stage Game Strategy for Multiclass Imbalanced Data Online Prediction

For multiclass imbalanced data online prediction, how to design a self-adapted model is a challenging problem. To address this issue, a novel dynamic multi-classification algorithm which uses two-stage game strategy has been put forward. Different from typical imbalanced classification methods, the proposed approach provided a self-updating model quantificationally, which can match the changes of arriving sample chunk automatically. In data generation phase, two dynamic ELMs with game theory are utilized for generating the lifelike minority class to equilibrate the distribution of different samples. In model update phase, both the current prediction performance and the cost sensitivity are taken into consideration simultaneously. According to the suffer loss and the shifty imbalance ratio, the proposed method develops the relationship between new weight and individual model, and an aggregate model of game theory is adopted to calculate the combination weight. These strategies help the algorithm reduce fitting error of sequence fragments. Also, alterative hidden-layer output matrix can be calculated according to the current fragment, thus building the steady network architecture in the next chunk. Numerical experiments are conducted on eight multiclass UCI datasets. The results demonstrate that the proposed algorithm not only has better generalization performance, but also improves the predictive ability of ELM method for minority samples.

On robust randomized neural networks for regression: a comprehensive review and evaluation

Data from real-world regression problems are quite often contaminated with outliers. In order to efficiently handle such undesirable samples, robust parameter estimation methods have been incorporated into randomized neural network (RNN) models, usually replacing the ordinary least squares (OLS) method. Despite recent successful applications to outlier-contaminated scenarios, significant issues remain unaddressed in the design of reliable outlier-robust RNN models for regression tasks. For example, the number of hidden neurons impacts directly on the norm of the estimated output weights, since the OLS method will rely on an ill-conditioned hidden-layer output matrix. Another design concern involves the high sensitivity of RNNs to the randomization of the hidden layer weights, an issue that can be suitably handled, e.g., by intrinsic plasticity techniques. Bearing these concerns in mind, we describe several ideas introduced in previous works concerning the design of RNN models that are both robust to outliers and numerically stable. A comprehensive evaluation of their performances is carried out across several benchmarking regression datasets taking into account accuracy, weight norms, and training time as figures of merit.

Multi-Objective Optimization for Flood Interval Prediction Based on Orthogonal Chaotic NSGA-II and Kernel Extreme Learning Machine

Deterministic flood prediction methods can only provide future point prediction results of the target variable. The intrinsic uncertainties and the fluctuation range of the prediction results cannot be evaluated. This study proposes a flood interval prediction method based on orthogonal chaotic non-dominated sorting genetic algorithm-II (OCNSGA-II) and kernel extreme learning machine (KELM) to estimate the uncertainty of the flood prediction results. The dual-output KELM model is exploited to predict the upper and lower bounds of the possible flood prediction result. The OCNSGA-II algorithm is employed to adjust the hidden layer output weights of the KELM model to minimize the prediction interval normalized average width (PINAW) and maximize the prediction interval coverage probability (PICP). The target variable with a disturbance of ±10% are taken as the initial upper and lower bounds. The superiority of the proposed method has been validated on one a real-world data set collected from the upper reaches of the Yangtze River in China. Results have shown that the proposed model can obtain prediction intervals with higher quality than the conventional single-objective interval prediction models and the other multi-objective benchmark models.

A Fast Algorithm for Multi-Class Learning from Label Proportions

Learning from label proportions (LLP) is a new kind of learning problem which has attracted wide interest in machine learning. Different from the well-known supervised learning, the training data of LLP is in the form of bags and only the proportion of each class in each bag is available. Actually, many modern applications can be successfully abstracted to this problem such as modeling voting behaviors and spam filtering. However, time-consuming training is still a challenge for LLP, which becomes a bottleneck especially when addressing large bags and bag sizes. In this paper, we propose a fast algorithm called multi-class learning from label proportions by extreme learning machine (LLP-ELM), which takes advantage of an extreme learning machine with fast learning speed to solve multi-class learning from label proportions. Firstly, we reshape the hidden layer output matrix and the training data target matrix of an extreme learning machine to adapt to the proportion information instead of the real labels. Secondly, a robust loss function with a regularization term is formulated and two efficient solutions are provided to different cases. Finally, various experiments demonstrate the significant speed-up of the proposed model with better accuracies on different datasets compared with several state-of-the-art methods.

Traffic speed prediction for urban transportation network: A path based deep learning approach

Traffic prediction, as an important part of intelligent transportation systems, plays a critical role in traffic state monitoring. While many studies accomplished traffic forecasting task with deep learning models, there is still an open issue of exploiting spatial-temporal traffic state features for better prediction performance, and the model interpretability has not been taken serious. In this study, we propose a path based deep learning framework which can produce better traffic speed prediction at a city wide scale, furthermore, the model is both rational and interpretable in the context of urban transportation. Specifically, we divide the road network into critical paths, which is helpful to mine the traffic flow mechanism. Then, each critical path is modeled through the bidirectional long short-term memory neural network (Bi-LSTM NN), and multiple Bi-LSTM layers are stacked to incorporate temporal information. At the stage of traffic prediction, the spatial-temporal features captured from these processes are fed into a fully-connected layer. Finally, results for each path are ensembled for network-wise traffic speed prediction. In the empirical studies, we compare the proposed model with multiple benchmark methods. Under a series of prediction scenarios (i.e., different input and prediction horizons), the superior performance of the proposed framework is validated. Moreover, by analyzing feature from hidden-layer output, the study explains the physical meaning of the hidden feature and illustrate model’s interpretability.

A proposed self-organizing radial basis function network for aero-engine thrust estimation

This paper proposes a new algorithm to construct self-organizing radial basis function neural networks (RBFNNs) for aero-engine thrust estimation. The algorithm can not only optimize centers and network size of the RBFNN but also automatically determine the connection weights. To reduce the dimensionality of particle and speed up the optimization process, spreads of an RBFNN are randomly initialized. Its weights are dynamically derived and adjusted by the product between the Moore–Penrose inverse of the hidden layer's outputs and the desired outputs. To optimize the centers and network size of the RBFNN, a strategy named multi-Gbest is adopted. Based on all these strategies, the proposed algorithm can effectively generate self-organizing RBFNNs with high accuracy. The successful application to aero-engine thrust estimation shows the practicability and effectiveness of the proposed algorithm.