Deep Feedforward Neural Network Research Articles

Hybrid deep fault detection and isolation: Combining deep neural networks and system performance models

With the increased availability of condition monitoring data on the one hand and the increased complexity of explicit system physics-based models on the other hand, the application of data-driven approaches for fault detection and isolation has recently grown. While detection accuracy of such approaches is generally very good, their performance on fault isolation often suffers from the fact that fault conditions affect a large portion of the measured signals thereby masking the fault source. To overcome this limitation, we propose a hybrid approach combining physical performance models with deep learning algorithms. Unobserved process variables are inferred with a physics-based performance model to enhance the input space of a data-driven diagnostics model. The resulting increased input space gains representation power enabling more accurate fault detection and isolation. To validate the effectiveness of the proposed method, we generate a condition monitoring dataset of an advanced gas turbine during flight conditions under healthy and four faulty operative conditions based on the Aero-Propulsion System Simulation (C-MAPSS) dynamical model. We evaluate the performance of the proposed hybrid methodology in combination with two different deep learning algorithms: deep feed forward neural networks and Variational Autoencoders, both of which demonstrate a significant improvement when applied within the hybrid fault detection and diagnostics framework. The proposed method is able to outperform pure data-driven solutions, particularly for systems with a high variability of operating conditions. It provides superior results both for fault detection as well as for fault isolation. For the fault isolation task, it overcomes the smearing effect that is commonly observed in pure data-driven approaches and enables a precise isolation of the affected signal. We also demonstrate that deep learning algorithms provide a better performance on the fault detection task compared to the traditional machine learning algorithms.

A feedforward deep neural network for predicting the state-of-charge of lithium-ion battery in electric vehicles

This study proposes a feedforward deep neural network to predict the parameters of the lithium-ion battery in electric vehicles. Correlation analysis is used to select the candidate parameters for the proposed model with no categorical variable. A direct feedforward deep artificial neural network is developed to predict a battery’s charge state and develop the proposed inverse model. The predicted state-of-charge of the direct model is combined with four virtual functions to form the input variables for the proposed inverse model. Furthermore, virtual functions are incorporated to enhance the predicting capability of the proposed inverse function model. The predicted multi-output variables of the proposed inverse model are speed, mileage, voltage, velocity, and state-of-charge. The proposed inverse model is superior to the feedforward deep neural network previously proposed in the literature because of its multiple output capabilities. Also, the proposed model makes decision-making easier when used for the design simulation than the single-output deep neural networks, which predict the state-of-charge of a battery only. The mean square error is used as the metric for accurate measurement. During the simulation by the proposed inverse model (with virtual functions), accuracy was 44.43 times higher than the traditional inverse deep neural network model. Redefined parameters were used to verify the findings of the model. This result suggests that incorporating virtual functions into a deep neural network model’s inverse model can improve the accuracy of battery and electric vehicle parameter predictions.

Experimental investigation of novel molten borate fuel cell supported by an artificial neural network for electrolyte composition selection

This research proposes idea of new kind of high temperature fuel cell – Molten Borate Fuel Cell (MBFC). The idea of new fuel cell is based on technical modification of well-known Molten Carbonate Fuel Cell by changing electrolyte composition to a borate-based mixture. The cell was tested in laboratory conditions as proof-of-concept investigation and reached OCV at around 1 V and 0.05 A/cm2 of current density. The optimization of borate-based electrolyte composition was conducted with a help of ANN and experimentally verified. The deep feedforward artificial neural network (DFF ANN) was proposed in this study to model the behavior of the new Molten Borate Fuel Cell. This modeling approach is well-known as a potent tool for dealing with complicated modeling and prediction tasks. This study presented many network designs for a range of operating circumstances. The ANN has been used to model and optimize the SOFC, MCFC, and PEMFC, but the design and performance of the MBFC were not previously investigated. The innovative type of fuel cells at issue - Molten Borate Fuel Cells - were designed and optimized using a deep learning algorithm. The most advanced model provides a dynamic forecast of fuel cell operation, taking thermal-flow and electrolyte material factors into account. With an average inaccuracy of 0.3%, all the models performed fair enough. An ANN-based technique may also be used to improve cell operating parameters. Moreover, the operating composition of a novel molten borate electrolyte might be improved in terms of electrochemical performance of the fuel cell.

Fluid catalytic cracking process quality-driven fault detection based on partial least squares and deep feedforward neural network

Fluid catalytic cracking (FCC) is an important process in petroleum processing. Effective monitoring of the status and quality of FCC is vital. Accurate description of the relationship between process and quality variables is the basis of quality-driven monitoring. Many process variables affect the quality of FCC; some of these effects are linear, and others are nonlinear. We propose a combination method from the perspective of linearity and nonlinearity to improve the monitoring performance of FCC quality. Partial least squares (PLS) is initially used to extract linear features, and its residual space is saved as the input of the deep feedforward neural network (DFNN). DFNN is then used to extract nonlinear features for the further decomposition of subspaces. The PLS-DFNN method accurately describes processes involving linearity and nonlinearity. We construct three monitoring statistics to characterize the types of faults. The proposed method proves its excellent effect on a numerical simulation data set. It effectively distinguishes the types of faults on the Tennessee Eastman process data set, and the fault detection rate is superior to other related methods. Finally, we apply this method to the actual FCC and verify the superiority of this combination.

Downward-Growing Neural Networks.

A major issue in the application of deep learning is the definition of a proper architecture for the learning machine at hand, in such a way that the model is neither excessively large (which results in overfitting the training data) nor too small (which limits the learning and modeling capabilities of the automatic learner). Facing this issue boosted the development of algorithms for automatically growing and pruning the architectures as part of the learning process. The paper introduces a novel approach to growing the architecture of deep neural networks, called downward-growing neural network (DGNN). The approach can be applied to arbitrary feed-forward deep neural networks. Groups of neurons that negatively affect the performance of the network are selected and grown with the aim of improving the learning and generalization capabilities of the resulting machine. The growing process is realized via replacement of these groups of neurons with sub-networks that are trained relying on ad hoc target propagation techniques. In so doing, the growth process takes place simultaneously in both the depth and width of the DGNN architecture. We assess empirically the effectiveness of the DGNN on several UCI datasets, where the DGNN significantly improves the average accuracy over a range of established deep neural network approaches and over two popular growing algorithms, namely, the AdaNet and the cascade correlation neural network.

Improved Adam-based Feedforward Deep Neural Network Model for Personalized Asthma Predictions

Improved Adam-based Feedforward Deep Neural Network Model for Personalized Asthma Predictions

Classification of handover interaction primitives in a COBOT–human context with a deep neural network

Object handover between humans and robots is an extremely important concern and lays on the edge of human–robot interaction challenges. This paper proposes a new approach to classify physical human interactions with a COBOT for handover operations. The human actions are categorized in simple interaction primitives. A deep neural network was devised and trained to classify torques and forces measured on the robot into one of four interaction primitives, namely pull, push, shake and twist. More specifically, an input vector of forces and torques observed and accumulated during a short span of half a second is fed into a properly trained feedforward deep neural network that classifies the interaction as one of the four primitives established. A specific dataset was created using several persons that provided abundant data both for training and testing. The results are very good not only in the training phase demonstrated by the convergence indicators, but also in the testing phase where interactions from previously unseen operators were successfully classified with an outstanding confidence.

Hybrid feature learning framework for the classification of encrypted network traffic

The purpose of traffic classification is to allocate bandwidth to different types of data on a network. Application-level traffic classification is important for identifying the applications that are in high demand on the network. Due to the increasing complexity and volume of internet traffic, machine learning and deep learning methods are being used more frequently in traffic classification. The focus of this research is to evaluate the performance of the Support Vector Machine (SVM) in classifying network packets by application type, as well as classifying the type of data communicated within an application. The research considers encrypted network packets, including those from Virtual Private Networks (VPN) and the WhatsApp mobile application. Previous research has shown that deep learning methods are effective in the feature learning process, so this study uses a simple feed-forward Deep Neural Network (DNN) to improve the performance of the SVM algorithm. Additionally, various feature learning frameworks based on deep learning, such as DNN, Autoencoder and PCA, are compared. The study concludes that the DNN is able to improve the F1 score of the SVM classifier from 0.78 to 0.90. Furthermore, the study shows that using a hybrid framework of DNN with SVM can address the class imbalance problem often present in machine learning.

Multiscale analysis of nonlinear systems using a hierarchy of deep neural networks

The application of multiscale methods that are based on computational homogenization, such as the well established FE2, remains in most cases a computationally challenging task. In a system that involves more than two length scales, the task of a FEN (N¿2) calculation becomes practically intractable if applied in the frame of conventional computational setups. This paper addresses this issue by introducing a novel surrogate model designed to accelerate the solution procedure of hierarchically formulated multiscale problems. The idea is to employ a sequence of deep feedforward neural networks (DNNs) that represent the hierarchy of the separate scales in the multiscale problem. Each DNN is trained to learn the constitutive law of a corresponding length scale of the problem. In a similar manner to the original problem, where each finer scale is contained in a coarser scale, DNNs representing fine scales are contained in the DNNs that represent coarser scales. At the end of the training process, a single deep network is produced which emulates the macroscopic behaviour by incorporating all physical mechanisms arising at each of the problem’s finer scales. This approach takes full advantage of the accuracy and modelling capabilities of the FEN schemes, while at the same time handles the immense computational requirements associated with their implementation. By virtue of the elaborated surrogate modelling setup, a four-scale structural system is solved using a FE4 scheme at reasonable computational times. In turn, this allows us to perform laborious sensitivity analyses in order to assess the uncertainty in the material parameters and its propagation to the macroscopic structural response.

Inverse of initial stress in carbon fiber reinforced polymer laminates using lamb waves and deep neural network

The prediction of the initial stress in composites is essential for the non-destructive testing (NDT) and structural health monitoring (SHM) of carbon fibre reinforced polymer (CFRP). This paper examines the potential of Lamb waves in the inverse of initial stress by calculating the influence of initial stress on the dispersion characteristics of Lamb waves propagating in multilayered CFRP laminates. By introducing the mechanics of incremental deformation into the linear three-dimensional elasticity theory, the Legendre orthogonal polynomial expansion (LOPE) method is used to mathematically model the Lamb wave propagating in multilayered CFRP laminates subjected to horizontal and vertical homogeneous initial stresses. Then, a three-hidden-layers Feed Forward Deep Neural Network (DNN) with Back Propagation (BP) algorithm is constructed to invert the magnitude and direction of the initial stresses. The input features are the phase velocities of fundamental Lamb wave A0 mode at five different frequencies. Both training and testing samples are obtained by LOPE forward calculation. An ablation experiment is presented to compare the two different activation functions. Finally, the accuracy of the inverse is verified by comparing with the available outcomes of LOPE forward calculation.

Precise geopressure predictions in active foreland basins: An application of deep feedforward neural networks

Precise geopressure predictions in active foreland basins: An application of deep feedforward neural networks

Short-Term Energy Load Forecasting Model with Sample Filtering

Short-term energy load forecasting is a crucial task in the power smart grid, which enables the power utilities to understand the future energy demands and plans to attain the demand and supply equilibrium, thereby optimizing power deployment and reducing power losses. Several techniques have been implemented to enhance energy load forecasting. However, the nonlinear nature of the data collected in the smart grid makes it difficult to attain 100% energy load forecasting accuracy. For instance, the Deep Feedforward Neural Networks model based on Input Attention Mechanism and Hidden Connection Mechanism has a mean absolute percentage error of 3.17%; model based on Sequence to Sequence Recurrent Neural Network with Attention had a mean absolute percentage error of 2.7%. The model based on Deep Recurrent Neural Networks with Levenberg–Marquardt backpropagation algorithm had a mean absolute percentage error of 0.58; and Deep Feedforward Neural Network with sample weights model had 3.22 % as root mean squared error. To improve energy load forecasting accuracy, this work proposed a model based on Deep Recurrent Neural Networks and sample filtering, which provides an exhaustive elucidation for modelling a sophisticated stochastic relationship between the input and output features. Deep Recurrent Neural Networks have proven to be good at modelling the nonlinearities in data of different fields and are mostly used in energy load forecasting to reduce forecasting error and a high degree of overfitting. Sample filtering is achieved through the use of K-Means clustering which determines the number of clusters to be used in the model. Findings from the study showed that by employing Deep Recurrent Neural Networks and sample filtering, the short-term energy load forecasting accuracy is improved in reference to mean absolute percentage error and root mean squared error of 0.31% and 1.014, respectively. As a result of the reduction in error, the energy demand and supply chain equilibrium are enhanced, thereby optimizing power deployment and reducing power losses.
 Keywords: Machine learning, Neural networks, Sample filtering, Smart grid, Short-term energy forecasting

Can a Computer “Learn” Nonlinear Chromatography?: Experimental Validation of Physics-Based Deep Neural Networks for the Simulation of Chromatographic Processes

This article presents the capabilities of machine learning in addressing the challenges related to the accurate description of adsorption equilibria in the design of chromatographic processes. Our previously developed physics-based artificial neural network framework for adsorption and chromatography emulation (PANACHE) approach is extended to simulate the dynamics of chromatographic columns without using adsorption isotherms. The incorporation of underlying conservation laws in the form of a physics-constrained loss function during training enables PANACHE models to infer the complex dynamics of chromatographic columns, even without the knowledge of adsorption isotherms. The isotherm-agnostic abilities of PANACHE models are tested by considering two binary systems with complex adsorption equilibria: 1) mixed Langmuir M1-type binary system and 2) Tröger’s base enantiomers on Chiralpak AD that show inflection in the isotherm. For each case study, unique feed-forward deep neural networks with an input layer, five hidden layers, and an output layer are trained for two solute components using the elution data either from simulations or experiments. The results show that PANACHE models are successful in predicting the dynamics of binary solute mixtures in chromatographic columns even in the absence of adsorption isotherms, with prediction errors in the order of 10–2 in the measure of relative L2-norm, for most of the cases. The ability of PANACHE models to infer the adsorption isotherms is also demonstrated.

Empirical Study of Overfitting in Deep Learning for Predicting Breast Cancer Metastasis

Overfitting may affect the accuracy of predicting future data because of weakened generalization. In this research, we used an electronic health records (EHR) dataset concerning breast cancer metastasis to study the overfitting of deep feedforward neural networks (FNNs) prediction models. We studied how each hyperparameter and some of the interesting pairs of hyperparameters were interacting to influence the model performance and overfitting. The 11 hyperparameters we studied were activate function, weight initializer, number of hidden layers, learning rate, momentum, decay, dropout rate, batch size, epochs, L1, and L2. Our results show that most of the single hyperparameters are either negatively or positively corrected with model prediction performance and overfitting. In particular, we found that overfitting overall tends to negatively correlate with learning rate, decay, batch size, and L2, but tends to positively correlate with momentum, epochs, and L1. According to our results, learning rate, decay, and batch size may have a more significant impact on both overfitting and prediction performance than most of the other hyperparameters, including L1, L2, and dropout rate, which were designed for minimizing overfitting. We also find some interesting interacting pairs of hyperparameters such as learning rate and momentum, learning rate and decay, and batch size and epochs.

DeepHAR: a deep feed-forward neural network algorithm for smart insole-based human activity recognition

Health monitoring, rehabilitation, and fitness are just a few domains where human activity recognition can be applied. In this study, a deep learning approach has been proposed to recognise ambulation and fitness activities from data collected by five participants using smart insoles. Smart insoles, consisting of pressure and inertial sensors, allowed for seamless data collection while minimising user discomfort, laying the baseline for the development of a monitoring and/or rehabilitation system for everyday life. The key objective has been to enhance the deep learning model performance through several techniques, including data segmentation with overlapping technique (2 s with 50% overlap), signal down-sampling by averaging contiguous samples, and a cost-sensitive re-weighting strategy for the loss function for handling the imbalanced dataset. The proposed solution achieved an Accuracy and F1-Score of 98.56% and 98.57%, respectively. The Sitting activities obtained the highest degree of recognition, closely followed by the Spinning Bike class, but fitness activities were recognised at a higher rate than ambulation activities. A comparative analysis was carried out both to determine the impact that pre-processing had on the proposed core architecture and to compare the proposed solution with existing state-of-the-art solutions. The results, in addition to demonstrating how deep learning solutions outperformed those of shallow machine learning, showed that in our solution the use of data pre-processing increased performance by about 2%, optimising the handling of the imbalanced dataset and allowing a relatively simple network to outperform more complex networks, reducing the computational impact required for such applications.

LiNLNet: Gauging required nonlinearity in deep neural networks

Feedforward deep neural networks (DNNs) commonly involve layer-wise linear operations and subsequent nonlinear operations, which are repeated through all layers. The nonlinear operations by nonlinear activations in each layer remarkably enhance the expressiveness of DNNs, resulting in the great success in a variety of application domains. Although the necessity of layer-wise nonlinear operations is agreed, the optimal nonlinearity for each layer in a given DNN is not clear. In this regard, we propose an easy-to-use method to layer-wise measure the optimal nonlinearity for a given DNN using its replica termed a linear-nonlinear network (LiNLNet). The key to the LiNLNet is the use of linear-nonlinear units (LiNLUs) whose degree of nonlinearity is parameterized by a trainable parameter p. The parameter p is shared among all LiNLUs in a given layer, thus indicating the layer-wise optimal nonlinearity. This method allows layer-level pruning such that the layers that do not require nonlinearity are merged into the subsequent layers, reducing computational complexity. For proofs of concept, we applied the proposed method to a MLP, AlexNet, VGG16, and ResNet18 on CIFAR-10 and ImageNet. The results commonly indicate the last hidden layer as a linear layer that may be merged into the output layer, reducing memory usage by 27% while maintaining the accuracy for LiNL-AlexNet on ImageNet.

Interpreting the effects of shale rock properties on seismic anisotropy by statistical and machine learning methods

The elastic anisotropy of unconventional shale reservoirs plays a crucial role in their economic hydrocarbon production. The strong anisotropy of organic-rich shale rocks, characterized by clay platelets, elongated pores, and organic material, presents challenges for traditional reservoir characterization methods, which are designed for isotropic or weakly anisotropic conventional sandstone reservoirs. To effectively describe the geomechanical behaviors of organic-rich shales and estimate changes in stress caused by hydraulic fracturing, it is necessary to develop practical methods for predicting anisotropy parameters from conventional rock property logs. This study proposes a statistical approach for predicting independent elastic stiffness coefficients for vertically transverse isotropic (VTI) shale from three shale rock properties: clay volume fraction, kerogen volume fraction, and total porosity. The results of this study show that the stiffness coefficients, C11 and C33, are related to P-wave anisotropy and porosity has a greater impact on P-wave anisotropy than clay and kerogen volume fractions. In contrast, changes in clay volume have a more sensitive effect on C44 and C66, providing S-wave anisotropy information that can help detect lithology changes. Furthermore, kerogen variation has a greater influence on both C12 and C13 than the effects of porosity and clay variations. From these relationships, we develop exponential nonlinear regression (ENLR) models for predicting independent elastic stiffness coefficients from three shale rock properties. These models are validated with the Midland Basin log data, and we improve the accuracy of the C13 estimation model using the deep feedforward neural network (DFNN) method. The results show that the statistical approach can help understand the effect of shale rock components on seismic anisotropy variations and better predict the variations in lithology, fluid, and organic contents in shales from anisotropic seismic data. Furthermore, the methodology can provide robust estimates of the stress changes from hydraulic fracturing, supporting the efficient and economically viable production of shale hydrocarbons.

Feed-Forward Deep Neural Network (FFDNN)-Based Deep Features for Static Malware Detection

The portable executable header (PEH) information is commonly used as a feature for malware detection systems to train and validate machine learning (ML) or deep learning (DL) classifiers. We propose to extract the deep features from the PEH information through hidden layers of a feed-forward deep neural network (FFDNN). The extraction of deep features of hidden layers represents the dataset with a better generalization for malware detection. While feeding the deep feature of one hidden layer to the succeeding layer, the Gaussian error linear unit (GeLU) activation function is applied. The FFDNN is trained with the GeLU activation function using the deep features of individual layers as well as concatenated deep features of all hidden layers. Similarly, the ML classifiers are also trained and validated in with individual layer deep features and concatenated features. Three highly effective ML classifiers, random forest (RF), support vector machine (SVM), and k-nearest neighbour (k-NN) have been investigated. The performance of the proposed model is demonstrated using a statically significant large dataset. The obtained results are interesting and encouraging in terms of classification accuracy. The classification accuracy reaches 99.15% with the internal discriminative deep feature for the proposed FFDNN-ML classifier with the GeLU activation function.

A hybrid deep learning - CFD approach for modeling nanoparticles’ sedimentation processes for possible application in clean energy systems

Sedimentation directly affects the thermal performance and efficiency of thermal systems such as boilers, heat exchangers, and solar collectors. This work investigates the effect of nanoparticles deposition inside a tube with possible application in parabolic solar collectors. This study combines the lattice Boltzmann (LBM) and the control finite volume (CFV) methods for a realistic simulation of nanoparticles deposition for the first time. While the bulk flow is solved using the CFV method, the flow behavior in the deposition layer is evaluated using the LBM model. Nanoparticle movements are also captured using dynamic mesh refinement in CFV in order to accurately predict their behavior. The numerical results are then used for training a deep feed-forward neural network with appropriate boundary conditions (DFNN-BC) to visualize and predict the transient sedimentation behavior. The prediction includes (i) representation of nanoparticles in the LB domain while it is trained during the particle movement in the FV domain and (ii) extension of the computational domain in space, which is three times bigger than the training domain. DFNN-BC is used to study the heat transfer and fluid flow characteristics for Reynolds numbers ranging from 12 to 50 where the working fluid is a nanofluid. The results indicated that using DFNN-BC can reduce the calculation time by 80% compared to the case where the entire domain is solved numerically. The results show that deposition has a maximum effect of 0.32% on the average velocity ratio (AVR) at Re = 12. This variation is related to the viscosity and shear stress of the fluid. With an increment in Reynolds number, the AVR decreases to 0.12%. This is because of the decrement in the number of sedimented nanoparticles. In addition, increasing the velocity significantly affects the rate of sedimentation and volume fraction ratio. It is also seen that the fluid's velocity and density increase by 8.69% and 6.53%, respectively, whereas the viscosity decreases by 7.74%. The findings of this study provide a better understanding of the details of the sedimentation process, such as particle behavior and variation in parameters near the surface, like concentration, thermal conductivity, and viscosity of the sedimentation and the formation of a deposition layer in fluid–particle multiphase flows. This, in turn, is expected to lead to cost savings in maintenance through more precise predictions of service periods for heat transfer equipment.

Intelligent Discrete Deep Learning Based Classification Methodology in Chemometrics.

In recent years, deep learning models have attracted much attention for classification purposes in chemometrics. The popularity of deep learning models in this field comes from their unique features like universal approximation capability with the desired accuracy. Deep learning classifiers use several intelligent processing layers to model mixed, complex, and nonlinear patterns in the underlying data sets, which is why the development of deep learning based models has never been stopped in the chemometrics literature. Despite the variety of deep learning classification models used in this field, they all use a continuous distance-based cost function in their learning processes. Although using a continuous cost function for learning deep classifiers is a common approach, it conflicts with the discrete nature of the classification problem. In fact, applying a continuous cost function for inherently discrete classification problems can reduce the performance of the classification. In this research, a novel discrete learning based classification approach is proposed and implemented on a deep feed-forward neural network as one of the most commonly used deep learning models to develop a different learning process for deep classification models. The basis of the proposed learning approach is maximizing a discrete matching function of the actual and fitted values instead of minimizing a continuous distance-based cost function. The proposed classification approach is evaluated on five benchmark data sets in the chemistry field. The empirical results indicated the superiority of the proposed discrete deep learning approach over its classic continuous form. The results of this study demonstrate the important effect of discrete learning processes on the performances of deep learning classification models. Therefore, the proposed methodology can be a powerful alternative to common classification approaches to analyze chemical data in the chemometrics field.