Number Of Hidden Layers Research Articles

Deep Neural Network Models for the Prediction of the Aggregate Base Course Compaction Parameters

Laboratory tests for the estimation of the compaction parameters, namely the maximum dry density (MDD) and optimum moisture content (OMC) are time-consuming and costly. Thus, this paper employs the artificial neural network technique for the prediction of the OMC and MDD for the aggregate base course from relatively easier index properties tests. The grain size distribution, plastic limit, and liquid limits are used as the inputs for the development of the ANNs. In this study, multiple ANNs (240 ANNs) are tested to choose the optimum ANN that produces the best predictions. This paper focuses on studying the impact of three different activation functions: number of hidden layers, number of neurons per hidden layer on the predictions, and heatmaps are generated to compare the performance of every ANN with different settings. Results show that the optimum ANN hyperparameters change depending on the predicted parameter. Additionally, the hyperbolic tangent activation is the most efficient activation function as it outperforms the other two activation functions. Additionally, the simplest ANN architectures results in the best predictions, as the performance of the ANNs deteriorates with the increase in the number of hidden layers or the number of neurons per hidden layers.

Machine Learning Modeling and Hyper-Parameter Optimization for Weld Nugget Formation and Failure Behavior of Resistance Spot Welds

The welding process parameters of resistance spot welding are determined by quality indicators, such as nugget diameter, and tensile shear test behavior, such as failure load and location. In this study, deep-learning models were investigated to predict the quality indicators from base materials and process parameter information. For each model, hyperparameters, such as the number of hidden layers, number of nodes in the hidden layer, learning rate of the optimizer, and number of epochs, were optimized based on the model performance. The regression models for nugget diameter and failure load showed coefficients of determination of 0.90 and 0.95, respectively. Two models were developed to classify failure location: a 1-step model that estimates the failure location from the base material information and process parameters, and a 2-step model that estimates the failure location from the base material information and the nugget diameter as predicted by the developed regression model. The classification models for failure location showed similar accuracies of approximately 90%.

Artificial neural network optimized by differential evolution for predicting diameters of jet grouted columns

A novel and effective artificial neural network (ANN) optimized using differential evolution (DE) is first introduced to provide a robust and reliable forecasting of jet grouted column diameters. The proposed computational method adopts the DE algorithm to tackle the difficulties in the training and performance of neural networks and optimize the four quintessential hyper-parameters (i.e. the epoch size, the number of neurons in a hidden layer, the number of hidden layers, and the regularization parameter) that govern the neural network efficacy. This approach is further enhanced by a stochastic gradient optimization algorithm to allow ‘expensive’ computation efforts. The ANN-DE is first trained using a prepared jet grouting dataset, then verified and compared with the prevalent machine learning tools, i.e. neural networks and support vector machine (SVM). The results show that, the ANN-DE outperforms the existing methods for predicting the diameter of jet grouting columns since it well balances training efficiency and model performance. Specifically, the ANN-DE achieved root mean square error (RMSE) values of 0.90603 and 0.92813 for the training and testing phases, respectively. The corresponding values were 0.8905 and 0.9006 for the optimized ANN, then, 0.87569 and 0.89968 for the optimized SVM, respectively. The proposed paradigm is bound to be useful for solving various geotechnical engineering problems regardless of multi-dimension and nonlinearity.

Mechanics-Guided optimization of an LSTM network for Real-Time modeling of Temperature-Induced deflection of a Cable-Stayed bridge

The main girder of cable-stayed bridges will experience significant vertical deflection under the effect of a temperature field. If the regression model based on the inputting temperature and outputting temperature-induced deflection can be established, then the reference value of the temperature-induced deflection on the cable-stayed bridge’s main girder can be obtained. The reference value has positive significance for bridge management and maintenance. However, because of the time lag between the temperature and temperature-induced deflection and the complexity on temperature features, the correlation between the temperature features and temperature-induced deflection shows a strong high-order nonlinearity. The output result of the temperature-induced deflection model obtained by linear regression has great amplitude error and phase error. The long short-term memory (LSTM) network, which belongs to deep learning, has strong nonlinear fitting performance and can consider time dependence. To temporally map the characteristics between temperature and temperature-induced deflection, a digital regression model of temperature-induced deflection based on an LSTM network is established. To make the LSTM network work well, it is necessary to extract the temperature features based on mechanical mechanism and input these temperature features to the digital regression model. At the same time, the architecture of the network must be optimized. For the LSTM network, one of the optimized parameters is the number of the cells in input layer, which determines the time length of the input data; the other is the number of hidden layers, which determines if the LSTM network can accommodate a larger amount of input information, that is, to obtain high generalization performance. Finally, the average temperature of the main girder, the vertical temperature difference of the main girder and the average temperature of the tower are taken as input temperature features. Driven by mechanical mechanism and deep learning, the output accuracy and stability of digital model built by the network with two LSTM hidden layers are exceedingly better than that of traditional linear regression model. The average error of the LSTM digital regression model is only 1.4%, and the maximum error is only 6%.

Channel State Estimation in LTE-Based Heterogenous Networks Using Deep Learning

Following the continuous development of the information technology, the concept of dense urban networks has evolved as well. The powerful tools, like machine learning, break new ground in smart network and interface design. In this paper the concept of using deep learning for estimating the radio channel parameters of the LTE (Long Term Evolution) radio interface is presented. It was proved that the deep learning approach provides a significant gain (almost 40%) with 10.7% compared to the linear model with the lowest RMSE (Root Mean Squared Error) 17.01%. The solution can be adopted as a part of the data allocation algorithm implemented in the telemetry devices equipped with the 4G radio interface, or, after the adjustment, the NB-IoT (Narrowband Internet of Things), to maximize the reliability of the services in harsh indoor or urban environments. Presented results also prove the existence of the inverse proportional dependence between the number of hidden layers and the number of historical samples in terms of the obtained RMSE. The increase of the historical data memory allows using models with fewer hidden layers while maintaining a comparable RMSE value for each scenario, which reduces the total computational cost.

OPTIMIZED DEEP NEURAL NETWORKS ARCHITECTURE MODEL FOR BREAST CANCER DIAGNOSIS

Breast cancer has increasingly claimed the lives of women. Oncologists use digital mammograms as a viable source to detect breast cancer and classify it into benign and malignant based on the severity. The performance of the traditional methods on breast cancer detection could not be improved beyond a certain point due to the limitations and scope of computing. Moreover, the constrained scope of image processing techniques in developing automated breast cancer detection systems has motivated the researchers to shift their focus towards Artificial Intelligence based models. The Neural Networks (NN) have exhibited greater scope for the development of automated medical image analysis systems with the highest degree of accuracy. As NN model enables the automated system to understand the feature of problem-solving without being explicitly programmed. The optimization for NN offers an additional payoff on accuracy, computational complexity, and time. As the scope and suitability of optimization methods are data-dependent, the choice of selection of an appropriate optimization method itself is emerging as a prominent domain of research. In this paper, Deep Neural Networks (DNN) with different optimizers and Learning rates were designed for the prediction of breast cancer and its classification. Comparative performance analysis of five distinct first-order gradient-based optimization techniques, namely, Adaptive Gradient (Adagrad), Root Mean Square Propagation (RMSProp), Adaptive Delta (Adadelta), Adaptive Moment Estimation (Adam), and Stochastic Gradient Descent (SGD), is carried out to make predictions on the classification of breast cancer masses. For this purpose, the Mammographic Mass dataset was chosen for experimentation. The parameters determined for experiments were chosen on the number of hidden layers and learning rate along with hyperparameter tuning. The impacts of those optimizers were tested on the NN with One Hidden Layer (NN1HL), DNN with Three Hidden Layers (DNN4HL), and DNN with Eight Hidden Layers (DNN8HL). The experimental results showed that DNN8HL-Adam (DNN8HL-AM) had produced the highest accuracy of 91% among its counterparts. This research endorsed that the incorporation of optimizers in DNN contributes to an increased accuracy and optimized architecture for automated system development using neural networks.

Non intrusive reduced order modeling of parametrized PDEs by kernel POD and neural networks

We propose a nonlinear reduced basis method for the efficient approximation of parametrized partial differential equations (PDEs), exploiting kernel proper orthogonal decomposition (KPOD) for the generation of a reduced order space and neural networks for the evaluation of the reduced order approximation. In particular, we use KPOD in place of the more classical POD, on a set of high-fidelity solutions of the problem at hand to extract a reduced basis. This method provides a more accurate approximation of the snapshots' set featuring a lower dimension, while maintaining the same efficiency as POD. A neural network (NN) is then used to find the coefficients of the reduced basis by following a supervised learning paradigm and shown to be effective in learning the map between the time/parameter values and the projection of the high-fidelity snapshots onto the reduced space. In this NN, both the number of hidden layers and the number of neurons vary according to the intrinsic dimension of the differential problem at hand and the size of the reduced space. This adaptively built NN attains good performances in both the learning and the testing phases. Our approach is then tested on two benchmark problems, a one-dimensional wave equation and a two-dimensional nonlinear lid-driven cavity problem. We finally compare the proposed KPOD-NN technique with a POD-NN strategy, showing that KPOD allows a reduction of the number of modes that must be retained to reach a given accuracy in the reduced basis approximation. For this reason, the NN built to find the coefficients of the KPOD expansion is smaller, easier and less computationally demanding to train than the one used in the POD-NN strategy.

Tuning Deep Neural Networks for Predicting Energy Consumption in Arid Climate Based on Buildings Characteristics

The dramatic growth in the number of buildings worldwide has led to an increase interest in predicting energy consumption, especially for the case of residential buildings. As the heating and cooling system highly affect the operation cost of buildings; it is worth investigating the development of models to predict the heating and cooling loads of buildings. In contrast to the majority of the existing related studies, which are based on historical energy consumption data, this study considers building characteristics, such as area and floor height, to develop prediction models of heating and cooling loads. In particular, this study proposes deep neural networks models based on several hyper-parameters: the number of hidden layers, the number of neurons in each layer, and the learning algorithm. The tuned models are constructed using a dataset generated with the Integrated Environmental Solutions Virtual Environment (IESVE) simulation software for the city of Buraydah city, the capital of the Qassim region in Saudi Arabia. The Qassim region was selected because of its harsh arid climate of extremely cold winters and hot summers, which means that lot of energy is used up for cooling and heating of residential buildings. Through model tuning, optimal parameters of deep learning models are determined using the following performance measures: Mean Square Error (MSE), Root Mean Square Error (RMSE), Regression (R) values, and coefficient of determination (R2). The results obtained with the five-layer deep neural network model, with 20 neurons in each layer and the Levenberg–Marquardt algorithm, outperformed the results of the other models with a lower number of layers. This model achieved MSE of 0.0075, RMSE 0.087, R and R2 both as high as 0.99 in predicting the heating load and MSE of 0.245, RMSE of 0.495, R and R2 both as high as 0.99 in predicting the cooling load. As the developed prediction models were based on buildings characteristics, the outcomes of the research may be relevant to architects at the pre-design stage of heating and cooling energy-efficient buildings.

Feasibility of Hybrid PSO-ANN Model for Identifying Soybean Diseases

Soybean disease has become one of vital factors restricting the sustainable development of high-yield and high-quality soybean industry. A hybrid artificial neural network (ANN) model optimized via particle swarm optimization (PSO) algorithm, which is denoted as PSO-ANN, is proposed in this paper for soybean diseases identification based on categorical feature inputs. Augmentation dataset is created via Synthetic minority over-sampling technique (SMOTE) to deal with quantitative insufficiency and categorical unbalance of the dataset. PSO algorithm is used to optimize the parameters in ANN, including the activation function, the number of hidden layers, the number of neurons in each hidden layer and the optimizer. In the end, ANN model with 2 hidden layers, 63 and 61 neurons in hidden layers respectively, Relu activation function and Adam optimizer yields the best overall test accuracy of 92.00%, compared with traditional machine learning methods. PSO-ANN shows superiority on various evaluation metrics, which may have great potential in crop diseases control for modern agriculture.

Evaluation of Neural Network Model for Estimating Pile Load Capacity

Neural network models based on deep learning algorithms are increasingly used for estimating pile load capacities as supplements of bearing capacity equations and field load tests. A series of hyperparameter tuning is required to improve the performance and reliability of developing a neural network model. In this study, the number of hidden layers and neurons, the activation functions, the optimizing algorithms of the gradient descent method, and the learning rates were tuned. The grid search method was applied for the tuning, which is a hyperpameter optimizer supplied by the developing platform. The cross-validation method was applied to enhance reliability for model validation. An appropriate number of epochs was determined using the early stopping method to prevent the overfitting of the model to the training data. The performance of the tuned optimum model evaluated for the test data set revealed that the model could estimate pile load capacities approximately with an average absolute error of 3,000 kN and a coefficient of determinant of 0.5.

Modeling and optimization of A-GTAW process using Box–Behnken design and hybrid BPNN-PSO approach

Gas tungsten arc welding (GTAW) is the most extensively used process capable of fabricating a wide range of alloys based on its distinctive merits has been introduced. However, some demerits have been reported among which shallow penetration is the most crucial ones. In order to deal with the poor penetration of the process, different procedures have been proposed among which activated gas tungsten arc welding (A-GTAW) is the most extensively used one. In this study effect of percentage of activating fluxes (TiO2 and SiO2) combination (F) and the most important process variables (welding current (C), welding speed (S)) on the most important process measures (weld bead width (WBW), depth of penetration (DOP), and consequently aspect ratio (ASR)) in welding of AISI316L austenite stainless steel parts have been investigated. Box-behnken design (BBD) has been used to design the experimental matrix required for date gathering, modeling, and statistical analysis purposes. A neural network with a back propagation algorithm (BPNN) in artificial neural network (ANN) modeling approach has been employed to relate the process input variables and output characteristics. The proper BPNN architecture (number of hidden layers and neurons/nodes in each hidden layer) has been determined using particle swarm optimization (PSO) algorithm. Moreover, process optimization in such a way that maximum DOP, minimum WBW, and desired ASR achieved has been carried out using PSO algorithm. Next, the performance of PSO algorithm has been checked using simulated annealing (SA) algorithm. Finally, to evaluate the performance of the proposed method a set of confirmation experimental test has been conducted. Results of experimental tests revealed that the proposed method is quite efficient in modeling and optimization (with less than 4% error) of A-GTAW process.

Stacked bidirectional LSTM RNN to evaluate the remaining useful life of supercapacitor

To predict the remaining useful life of supercapacitor, a data-based model is established by using a stacked bidirectional long short-term memory recurrent neural network. On the basis of the traditional long short-term memory recurrent neural network, a reverse recurrent layer with t time and subsequent time values in the input sequence is added. A stacked network can ensure enough capacity space. Simulation results show that the network has superior performance when the number of hidden layers is 2, the predicted RMSE and MAE are 0.0275 and 0.0241, respectively. Meanwhile, simulation compares ordinary and bidirectional recurrent neural networks and the bidirectional recurrent neural networks with different recurrent units. For subsequent ameliorate, this project will add swarm intelligence algorithm to optimize the initial weight of neural network and reduce the initial prediction error.

Investigating deep feedforward neural networks for classification of transposon-derived piRNAs

PIWI-interacting RNAs (piRNAS) form an important class of non-coding RNAs that play a key role in gene expression regulation and genome integrity by silencing transposable elements. However, despite the importance of piRNAs and the large application of deep learning in computational biology, there are few studies of deep learning for piRNAs prediction. Still, current methods focus on using advanced architectures like CNN and variations. This paper presents an investigation on deep feedforward network models for classification of human transposon-derived piRNAs. We developed a lightweight predictor (when compared to other deep learning methods) and we show by practical evidence that simple neural networks can perform as well as better than complex neural networks when using the appropriate hyperparameters. For that, we train, analyze and compare the results of a multilayer perceptron with different hyperparameter choices, such as numbers of hidden layers, activation functions and optimizers, clarifying the advantages and disadvantages of each choice. Our proposed predictor reached a F-score of 0.872, outperforming other state-of-the-art methods for human transposon-derived piRNAs classification. In addition, to better access the generalization of our proposal, we also showed it achieved competitive results when classifying piRNAs of other species.

A Neural Network Approach towards Generalized Resistive Switching Modelling.

Resistive switching behaviour has been demonstrated to be a common characteristic to many materials. In this regard, research teams to date have produced a plethora of different devices exhibiting diverse behaviour, but when system design is considered, finding a ‘one-model-fits-all’ solution can be quite difficult, or even impossible. However, it is in the interest of the community to achieve more general modelling tools for design that allows a quick model update as devices evolve. Laying the grounds with such a principle, this paper presents an artificial neural network learning approach to resistive switching modelling. The efficacy of the method is demonstrated firstly with two simulated devices and secondly with a amorphous IGZO device. For the amorphous IGZO device, a normalized root-mean-squared error (NRMSE) of 5.66 × 10−3 is achieved with a network structure, representing a good balance between model complexity and accuracy. A brief study on the number of hidden layers and neurons and its effect on network performance is also conducted with the best NRMSE reported at 4.63 × 10−3. The low error rate achieved in both simulated and real-world devices is a good indicator that the presented approach is flexible and can suit multiple device types.

Comparison between Artificial Neural Network and Rigorous Mathematical Model in Simulation of Industrial Heavy Naphtha Reforming Process

In this study, an artificial neural network (ANN) model was developed and compared with a rigorous mathematical model (RMM) to estimate the performance of an industrial heavy naphtha reforming process. The ANN model, represented by a multilayer feed forward neural network (MFFNN), had (36-10-10-10-34) topology, while the RMM involved solving 34 ordinary differential equations (ODEs) (32 mass balance, 1 heat balance and 1 momentum balance) to predict compositions, temperature, and pressure distributions within the reforming process. All computations and predictions were performed using MATLAB® software version 2015a. The ANN topology had minimum MSE when the number of hidden layers, number of neurons in the hidden layer, and the number of training epochs were 3, 10, and 100,000, respectively. Extensive error analysis between the experimental data and the predicted values were conducted using the following error functions: coefficient of determination (R2), mean absolute error (MAE), mean relative error (MRE), and mean square error (MSE). The results revealed that the ANN (R2 = 0.9403, MAE = 0.0062) simulated the industrial heavy naphtha reforming process slightly better than the rigorous mathematical model (R2 = 0.9318, MAE = 0.007). Moreover, the computational time was obviously reduced from 120 s for the RMM to 18.3 s for the ANN. However, one disadvantage of the ANN model is that it cannot be used to predict the process performance in the internal points of reactors, while the RMM predicted the internal temperatures, pressures and weight fractions very well.

Machine Learning Approach for Modeling and Control of a Commercial Heliocentris FC50 PEM Fuel Cell System

In recent years, machine learning (ML) has received growing attention and it has been used in a wide range of applications. However, the ML application in renewable energies systems such as fuel cells is still limited. In this paper, a prognostic framework based on artificial neural network (ANN) is designed to predict the performance of proton exchange membrane (PEM) fuel cell system, aiming to investigate the effect of temperature and humidity on the stack characteristics and on tracking control improvements. A large part of the experimental database for various operating conditions has been used in the training operation to achieve an accurate model. Extensive tests with various ANN parameters such as number of neurons, number of hidden layers, selection of training dataset, etc., are performed to obtain the best fit in terms of prediction accuracy. The effect of temperature and humidity based on the predicted model are investigated and compared to the ones obtained from real-time experiments. The control design based on the predicted model is performed to keep the stack operating point at an adequate power stage with high-performance tracking. Experimental results have demonstrated the effectiveness of the proposed model for performance improvements of PEM fuel cell system.

Modeling and optimization of A-GTAW process using back propagation neural network and heuristic algorithms

Apart from different merits of using conventional gas tungsten arc welding (C-GTAW) process, some demerits have been introduced among which shallow penetration is the most important ones. In order to cope with the mentioned disadvantage, some procedures have been proposed among which using a paste like coating of activating flux during welding process known as activated-GTAW (A-GTAW) is the most extensively used ones. In this study effect of the most important process variables (welding current (C), welding speed (S)) and percentage of activating fluxes (TiO2 and SiO2) combination (F) on the most important quality characteristics (depth of penetration (DOP), weld bead width (WBW), and consequently aspect ratio (ASR)) in welding of AISI316L austenite stainless steel parts have been considered. To gather the required data for modeling and optimization purposes, box-behnken design (BBD) in design of experiments (DOE) approach has been used. In order to establish a relation between process input variables and output characteristics, back propagation neural network (BPNN) has been employed results of which have been compared with regression modeling outputs. Particle swarm optimization (PSO) algorithm has been used for determination of BPNN architecture (number of hidden layers and neurons/nodes in each hidden layer). Dragonfly (DFA) and PSO algorithms have been employed for process optimization in such a way that desired AR, minimum WBW, and maximum DOP achieved simultaneously. Finally, confirmation experimental tests have been carried out to evaluate the performance of the proposed method. Based on the results, the proposed procedure is efficient in modeling and optimization (with less than 3% error) of A-GTAW process.

Radar target shape recognition using a gated recurrent unit based on RCS time series' statistical features by sliding window segmentation

The radar cross section (RCS) is an important parameter that reflects the scattering characteristics of radar targets. Based on the monostatic radar RCS time series' statistical features by sliding window segmentation, a novel sliding window-statistical-gated recurrent unit (SW-S-GRU) method for radar target recognition (RTR) is proposed using a GRU. The sliding window method divides the RCS time series into many segments for obtaining the statistical features as the input of the GRU. The deeper features are extracted from the segmented time series and then classified by the GRU. Under the model of precession motion, simulation results show that the proposed method achieves a great recognition accuracy. Even under the condition of a low SNR, the proposed method can obtain better recognition accuracy. For the precession dataset, the effect of sliding window parameters on RTR is analysed. The simulation results show that the recognition result of the model is better using the longer series overlap length between adjacent times. And when the number of hidden layers is 8 or above 8, the recognition accuracy does not change. Using the publicly available measured drone RCS data, the proposed method can also identify the drone well.

On the rate of convergence of fully connected deep neural network regression estimates

Recent results in nonparametric regression show that deep learning, that is, neural network estimates with many hidden layers, are able to circumvent the so-called curse of dimensionality in case that suitable restrictions on the structure of the regression function hold. One key feature of the neural networks used in these results is that their network architecture has a further constraint, namely the network sparsity. In this paper, we show that we can get similar results also for least squares estimates based on simple fully connected neural networks with ReLU activation functions. Here, either the number of neurons per hidden layer is fixed and the number of hidden layers tends to infinity suitably fast for sample size tending to infinity, or the number of hidden layers is bounded by some logarithmic factor in the sample size and the number of neurons per hidden layer tends to infinity suitably fast for sample size tending to infinity. The proof is based on new approximation results concerning deep neural networks.

Design of Neural Networks

Artificial Neural Networks (ANN) and Deep Learning (DL) are used to solve complex problems including image recognition, speech recognition and have applications in new technologies for autonomous driving, facial recognition, detecting cancers from imaging samples among others. Various design considerations are involved in the design, training, and testing of Artificial Neural Networks (ANN). These include the design of the input/output layer, the structure and number of hidden layers, the data/data-structures of variables, the transformative functions embedded in the network, the optimizers being considered, the learning rate and its systematic adjustment, the prudent usage of dropout, the parallelism-related batch-size, the number of epochs, the adaptive logic for systematically changing the network for better fit, etc. While all these methods and techniques are sensible and relevant, there lacks an overarching framework for the needed design. This paper considers the design of an ANN from an Axiomatic Design (AD) perspective that parallels the biological inspiration for ANN’s in the first place, i.e., the brain. The axiomatic design approach is used for explicating and extricating the form, function, and adaptive evolution of the underlying network.