Although Artificial Neural Networks Research Articles

Artificial neural networks for performance prediction of full-scale wastewater treatment plants: a systematic review.

Wastewater treatment plants (WWTPs) are complex systems that must maintain high levels of performance to achieve adequate effluent quality to protect the environment and public health. Artificial intelligence and machine learning methods have gained attention in recent years for modeling complex problems, such as wastewater treatment. Although artificial neural networks (ANNs) have been identified as the most common of these methods, no study has investigated the development and configuration of these models. We conducted a systematic literature review on the use of ANNs to predict the effluent quality and removal efficiencies of full-scale WWTPs. Three databases were searched, and 44 records of the 667 identified were selected based on the eligibility criteria. The data extracted from the papers showed that the majority of studies used the feedforward neural network model with a backpropagation training algorithm to predict the effluent quality of plants, particularly in terms of organic matter indicators. The findings of this research may help in the search for an optimum design modeling process for future studies of similar prediction problems.

Determination of the effect of valonia tannin when used as a filler on the formaldehyde emission and adhesion properties of plywood with artificial neural network analysis

Although artificial neural networks (ANN) have been used frequently in engineering applications, it has been determined that they are not preferred much, especially in determining adhesive properties and usage. Hence, this study aimed to determine the effect of urea formaldehyde (UF) resin mixed with valonia tannin obtained from Turkish oak acorns (Quercus aegilops) as a filler on the formaldehyde emission and bonding strength of plywood panels by using ANN modelling. For this purpose, six different valonia tannin contents and three different adhesive usages were chosen, and they were compared to control panels. The formaldehyde emission and bonding strength experimental data obtained according to relevant standards were analysed by using ANN. The prediction models with the best performance and acceptable deviations were determined using statistical and graphical comparisons between the experimental data and the prediction values obtained from the ANN analysis. Using these prediction models, the formaldehyde emission and bonding strength data were obtained according to the amount of valonia tannin and adhesive usage not used in the experimental study. Experimental results have proved that valonia tannin obtained from Turkish oak acorns is a good formaldehyde scavenger for UF adhesives, but it has been determined that it has a negative effect on the bonding strength of plywood. As a result of the ANN analysis, the lowest formaldehyde emission values were obtained from 12% to 15% valonia tannin and 140 g/m2 - 146 g/m2 adhesive usage. The bonding strength values of all the predicted plywood groups exceeded the limit value of the relevant standard except for the 5% valonia tannin and 140 g/m2 - 158 g/m2 adhesive usage.

Interpretable artificial neural networks for retrospective QbD of pharmaceutical tablet manufacturing based on a pilot-scale developmental dataset

As the pharmaceutical industry increasingly adopts the Pharma 4.0. concept, there is a growing need to effectively predict the product quality based on manufacturing or in-process data. Although artificial neural networks (ANNs) have emerged as powerful tools in data-rich environments, their implementation in pharmaceutical manufacturing is hindered by their black-box nature. In this work, ANNs were developed and interpreted to demonstrate their applicability to increase process understanding by retrospective analysis of developmental or manufacturing data. The in vitro dissolution and hardness of extended-release, directly compressed tablets were predicted from manufacturing and spectroscopic data of pilot-scale development. The ANNs using material attributes and operational parameters provided better results than using NIR or Raman spectra as predictors. ANNs were interpreted by sensitivity analysis, helping to identify the root cause of the batch-to-batch variability, e.g., the variability in particle size, grade, or substitution of the hydroxypropyl methylcellulose excipient. An ANN-based control strategy was also successfully utilized to mitigate the batch-to-batch variability by flexibly operating the tableting process. The presented methodology can be adapted to arbitrary data-rich manufacturing steps from active substance synthesis to formulation to predict the quality from manufacturing or development data and gain process understanding and consistent product quality.

Learning the Indicative Patterns of Simulated Force Changes in Soil Moisture by BP Neural Networks and Finding Differences with SMAP Observations

Soil moisture is a vital land surface variable that can influence climate change. Many problems in soil moisture data require the identification of signals obscured by anthropogenic external forces (including greenhouse gases such as CO2 and aerosol radiative force), natural forces (such as volcanic and solar activity), and internal variability (such as ENSO, NAO, and PDO). Although artificial neural networks (ANNs) have been widely studied in making accurate predictions, the studies of interpretation of ANNs in soil moisture are still rare. Hence, the proposed method aims to assist in the study of interpretating soil moisture data. Specifically, first, an ANN model is trained to predict the approximate year of the simulations by identifying the spatial patterns of qualitative changes in soil moisture. After accurately predicting the approximate year, the spatial patterns in the ANN model, acting as “reliable indicators” of the force changes, are the different natures of regional signals. Then, the simulated data and Soil Moisture Active and Passive (SMAP) observations are fed into the trained ANN separately, and the specific differences are observed by the Deep Taylor Decomposition (DTD) visualization tool. By comparing with the standard multiple linear regression method, the results of the ANN model can provide the reliable indicators of change for a specific year, thus providing meaningful information from the ANN model according to the common soil moisture data. The results show that a large correlation exists between eastern Asia and western North America during the 21st century, and the correlation increases with time in Australia. This also reflects the strong force signal due to a combination of anthropogenic and external forces that has played a role in soil moisture over the decades and can clearly discern the differences between model simulations and observed data. This study indicates that the proposed method using ANNs and visualization tools enables relatively accurate predictions and the discovery of unknown patterns within soil moisture data.

Fine-Grained Power Modeling of Multicore Processors Using FFNNs

To minimize power consumption while maximizing performance, today’s multicore processors rely on fine-grained run-time dynamic power information—both in the time domain, e.g. mu s to ms, and space domain, e.g. core-level. The state-of-the-art for deriving such power information is mainly based on predetermined power models which use linear modeling techniques to determine the core-performance/core-power relationship. However, with multicore processors becoming ever more complex, linear modeling techniques cannot capture all possible core-performance related power states anymore. Although artificial neural networks (ANN) have been proposed for coarse-grained power modeling of servers with time resolutions in the range of seconds, few works have yet investigated fine-grained ANN-based power modeling. In this paper, we explore feed-forward neural networks (FFNNs) for core-level power modeling with estimation rates in the range of 10 kHz. To achieve a high estimation accuracy while minimizing run-time overhead, we propose a multi-objective-optimization of the neural architecture using NSGA-II with the FFNNs being trained on performance counter and power data from a complex-out-of-order processor architecture. We show that relative power estimation error for the highest accuracy FFNN decreases on average by 7.5% compared to a state-of-the-art linear power modeling approach and decreases by 5.5% compared to a multivariate polynomial regression model. For the FFNNs optimized for both accuracy and overhead, the average error decreases between 4.1% and 6.7% compared to linear modeling while offering significantly lower overhead compared to the highest accuracy FFNN. Furthermore, we propose a micro-controller-based and an accelerator-based implementation for run-time inference of the power modeling FFNN and show that the area overhead is negligible.

Artificial Intelligence and Statistical Techniques in Short-Term Load Forecasting: a Review

Electrical utilities depend on short-term demand forecasting to adjust proactively the production and the distribution in anticipation of major variations. This systematic review analyzes 240 works published in scholarly journals between 2000 and 2019 that focus on applying Artificial Intelligence (AI), statistical, and hybrid models to Short-Term Load Forecasting (STLF). This work represents the most comprehensive review of works on this subject to date. A complete analysis of the literature is conducted in order to identify the most popular and accurate techniques as well as existing gaps. The findings show that although Artificial Neural Networks (ANN) continue to be the most commonly used standalone technique, researchers have been exceedingly opting for hybrid combinations of different techniques to leverage the combined advantages of individual methods. The review demonstrates that it is commonly possible with these hybrid combinations to achieve prediction accuracy exceeding 99%. The most successful duration for short-term forecasting has been identified as prediction for a duration of one day at an hourly interval. The review has identified a deficiency in access to datasets needed for training of the models. A significant gap has been identified in researching regions other than Asia, Europe, North America, and Australia.

Toward a Digital Twin: Time Series Prediction Based on a Hybrid Ensemble Empirical Mode Decomposition and BO-LSTM Neural Networks

Abstract Precise time series prediction serves as an important role in constructing a digital twin (DT). The various internal and external interferences result in highly nonlinear and stochastic time series. Although artificial neural networks (ANNs) are often used to forecast time series because of their strong self-learning and nonlinear fitting capabilities, it is a challenging and time-consuming task to obtain the optimal ANN architecture. This paper proposes a hybrid time series prediction model based on an ensemble empirical mode decomposition (EEMD), long short-term memory (LSTM) neural networks, and Bayesian optimization (BO). To improve the predictability of stochastic and nonstationary time series, the EEMD method is implemented to decompose the original time series into several components (each component is a single-frequency and stationary signal) and a residual signal. The decomposed signals are used to train the neural networks, in which the hyperparameters are fine-tuned by the BO algorithm. The following time series data are predicted by summating all the predictions of the decomposed signals based on the trained neural networks. To evaluate the performance of the proposed EEMD-BO-LSTM neural networks, this paper conducts two case studies (the wind speed prediction and the wave height prediction) and implements a comprehensive comparison between the proposed method and other approaches including the persistence model, autoregressive integrated moving average (ARIMA) model, LSTM neural networks, BO-LSTM neural networks, and EEMD-LSTM neural networks. The results show an improved prediction accuracy using the proposed method by multiple accuracy metrics.

Model forecasting based on two-stage feature selection procedure using orthogonal greedy algorithm

Currently, forecasting and feature selection tasks are attracting considerable attention from various scientific fields including global solar radiation forecasting, signal processing, microarray data analysis, finance, medicine and others. However, both selection inconsistency and the intractable computational cost pose critical difficulties when implementing forecasting tasks. Although artificial neural networks (ANNs) are useful for forecasting, a large number of nuisance features are employed. To establish an interpretable forecasting model, feature selection techniques are combined with ANNs to reduce the number of inputs and the complexity of network structures. However, these approaches shrink the estimate, which results in inaccurate forecasting results. To overcome these drawbacks, this paper successfully investigates a novel soft computing approach referred to as a two-stage feature selection procedure using the orthogonal greedy algorithm (TSOGA) to select the important features as inputs of ANNs. A simple-to-implement and efficient computational algorithm is designed, and the theoretical analysis is also provided. Furthermore, the high dimensional Bayesian information criterion (HDBIC) is utilized to select the optimal forecasting model. Real data experiments directly demonstrate the outstanding forecasting performances of the proposed TSOGA approach compared with other state-of-the-art techniques.

Artificial neural networks for the cost estimation of stamping dies

Although artificial neural networks (ANN) are more known in the field of image recognition and forecasting, cost estimation has become another emerging area in recent years. In this study, the establishment of an intelligent system was attempted for forecasting the total cost of sheet metal stamping dies. In this context, where the cost of stamping dies is estimated with a conventional approach which has been applied in the company up to now, the ANN and multiple regression analysis and the performance of the three cost-estimation models are examined. The examinations are based on the data of previous costs and use a number of critical criteria which are decided by experienced tool makers and engineers from every level of the organization of the seven companies which produce stamping dies. The comparative study reveals that the ANN system outperforms the traditional linear regression analysis model and conventional approach used for cost estimation. Thus, it is possible for firms which produce stamping dies to obtain a fairly accurate prediction with an ANN model and determined criteria.

A New Data Mining Scheme Using Artificial Neural Networks

Classification is one of the data mining problems receiving enormous attention in the database community. Although artificial neural networks (ANNs) have been successfully applied in a wide range of machine learning applications, they are however often regarded as black boxes, i.e., their predictions cannot be explained. To enhance the explanation of ANNs, a novel algorithm to extract symbolic rules from ANNs has been proposed in this paper. ANN methods have not been effectively utilized for data mining tasks because how the classifications were made is not explicitly stated as symbolic rules that are suitable for verification or interpretation by human experts. With the proposed approach, concise symbolic rules with high accuracy, that are easily explainable, can be extracted from the trained ANNs. Extracted rules are comparable with other methods in terms of number of rules, average number of conditions for a rule, and the accuracy. The effectiveness of the proposed approach is clearly demonstrated by the experimental results on a set of benchmark data mining classification problems.

Improving weighted information criterion by using optimization

Although artificial neural networks (ANN) have been widely used in forecasting time series, the determination of the best model is still a problem that has been studied a lot. Various approaches available in the literature have been proposed in order to select the best model for forecasting in ANN in recent years. One of these approaches is to use a model selection strategy based on the weighted information criterion (WIC). WIC is calculated by summing weighted different selection criteria which measure the forecasting accuracy of an ANN model in different ways. In the calculation of WIC, the weights of different selection criteria are determined heuristically. In this study, these weights are calculated by using optimization in order to obtain a more consistent criterion. Four real time series are analyzed in order to show the efficiency of the improved WIC. When the weights are determined based on the optimization, it is obviously seen that the improved WIC produces better results.

Dimensionality Reduction in Data Mining Using Artificial Neural Networks

The use of classic dimension reduction techniques can be considered customary practice within the context of data mining (DM). Nevertheless, although artificial neural networks (ANNs) are one of the most important DM techniques, specific ANN architectures for dimensionality reduction, such as the principal components analysis ANN (PCA-ANN) and the linear auto-associative ANN (LA-ANN), are used on far fewer occasions. In this study, categorical principal component analysis (CATPCA) and the two ANN procedures are studied and compared searching for uniqueness in an applied context relative to personality variables and drug consumption. A sample of 7,030 adolescents completed a personality test made up of 20 dichotomous items with a hypothesized four-factor latent model. Results point out that both ANN factor solutions converge to those obtained using CATPCA. Nevertheless, possible drawbacks of the ANN techniques lie in their relatively complex application, as well as in the need to use visual graphic analysis as a support for interpreting the factorized solutions.

Structural optimisation and input selection of an artificial neural network for river level prediction

Summary Accurate river level prediction, necessary for reliable flood forecasting, is a difficult task due to the complexity and inherent nonlinearity of the catchment hydrological system. Although artificial neural networks (ANNs) offer advantages over mechanistic or conceptual hydrological models for river level prediction, their applicability is limited by the fact that each ANN has to be specifically optimised and trained for a particular prediction problem and suitable input vectors selected. A recently developed novel optimisation algorithm combining properties of simulated annealing and tabu search is used to arrive at an optimal ANN for the prediction of river levels 5 h in advance. The algorithm seeks to minimise the value of a cost function based on the complexity and performance of the ANN. This is done by removing inter-neuron connections and adjusting the weights of the remaining connections. The candidate inputs presented to the algorithm were: current values of river levels at the flood location and two upstream locations; the change in level over the previous 4 h at the flood point, mean sea level pressure (SLP) and the change in SLP over the previous 24 h. The optimisation removed 79% of the network connections and three of the candidate inputs, leaving the current levels at the two upstream locations and at the flood point as the only inputs. The optimised ANN was then trained using the standard backpropagation algorithm. This methodology produces an ANN of greatly reduced complexity albeit with a reduced performance compared to an unoptimised ANN trained with backpropagation only. However, it has the advantage of being generally applicable and represents an improvement over trial and error as a method of ANN structural optimisation and input selection. For this prediction problem, current levels at two upstream locations and at the flood point are the best predictors of the level at the flood point.

Evolution of neural networks for classification and regression

Although Artificial Neural Networks (ANNs) are important data mining techniques, the search for the optimal ANN is a challenging task: the ANN should learn the input–output mapping without overfitting the data and training algorithms may get trapped in local minima. The use of Evolutionary Computation (EC) is a promising alternative for ANN optimization. This work presents two hybrid EC/ANN algorithms: the first evolves neural topologies while the latter performs simultaneous optimization of architectures and weights. Sixteen real-world tasks were used to test these strategies. Competitive results were achieved when compared with a heuristic model selection and other Data Mining algorithms.

Estimating Saturated Hydraulic Conductivity In Spatially Variable Fields Using Neural Network Ensembles

Modeling contaminant and water flow through soil requires accurate estimates of soil hydraulic properties in field scale. Although artificial neural networks (ANNs) based pedotransfer functions (PTFs) have been successfully adopted in modeling soil hydraulic properties at larger scales (national, continental, and intercontinental), the utility of ANNs in modeling saturated hydraulic conductivity (Ks) at a smaller (field) scale has rarely been reported. Hence, the objectives of this study are (i) to investigate the applicability of neural networks in estimating Ks at field scales, (ii) to compare the performance of the field‐scale PTFs with the published neural networks program Rosetta, and (iii) to compare the performance of two different ensemble methods, namely Bagging and Boosting in estimating Ks Datasets from two distinct sites are considered in the study. The performances of the models were evaluated when only sand, silt, and clay content (SSC) were used as inputs, and when SSC and bulk density ρb (SSC+ ρb) were used as inputs. For both datasets, the field scale models performed better than Rosetta The comparison of field‐scale ANN models employing bagging and boosting algorithms indicates that the neural network model employing the boosting algorithm results in better generalization by reducing both the bias and variance of the neural network models.

Calibration and validation of neural networks to ensure physically plausible hydrological modeling

Although artificial neural networks (ANNs) have proven to be useful tools for modeling many aspects of the hydrological cycle, the fact that they do not provide any means of exploiting fundamental knowledge of the system means that they are still viewed with some skepticism. In this paper, an approach is presented for incorporating information about relative input contributions in the development of an ANN during the calibration and validation stages. Two case studies are presented which highlight the uncertainty associated with calibrating and validating an ANN based on predictive error alone and demonstrates the necessity of constraining the calibration of an ANN to ensure physical plausibility. The proposed technique was used in the comparison of three training algorithms in terms of their ability to find a globally optimal solution and it was identified that neither in-sample nor out-of-sample performance measures are very informative about the solutions obtained, nor do they necessarily indicate that a reasonable approximation of the underlying relationship has been achieved. It was shown that by applying constraints to the objective function, an ANN could be developed with physically plausible input contributions and comparable predictive performance to that of an unconstrained model. A sensitivity analysis was carried out to verify the proposed methodology.

Studying the Explanatory Capacity of Artificial Neural Networks for Understanding Environmental Chemical Quantitative Structure−Activity Relationship Models

Although artificial neural networks (ANNs) have been shown to exhibit superior predictive power in the study of quantitative structure-activity relationships (QSARs), they have also been labeled a "black box" because they provide little explanatory insight into the relative influence of the independent variables in the predictive process so that little information on how and why compounds work can be obtained. Here, we have turned our interests to their explanatory capacities; therefore, a method was proposed for assessing the relative importance of variables indicating molecular structure, on the basis of axon connection weights and partial derivatives of the ANN output with respect to its input, which can identify variables that significantly contribute to network predictions, and providing a variable selection method for ANNs. We show that, by extending this approach to ANNs, the "black box" mechanics of ANNs can be greatly illuminated, thereby making it very useful in understanding environmental chemical QSAR models.

A visual method for determining variable importance in an artificial neural network model: An empirical benchmark study

Although artificial neural networks (ANNs) are an established marketing support tool, they are criticised for their inability to explain their results. The purpose of this study is to demonstrate a visual approach to ANN variable interpretation. Scanner data are used to build a well-known choice model, first as a multinomial logit and secondly as an ANN. Response elasticity graphs are then built for each ANN model variable. These graphs are used to interpret the model variables and are benchmarked against the t-statistics of the logit. The results suggest that modellers using this visual ANN approach can obtain a result that aids in variable interpretation and possibly provides a richer understanding of model behaviour than a standard statistical methodology.

Neural Network and Perturbation Theory Hybrid Models for Eigenvalue Prediction

Although artificial neural networks (ANNs) are powerful tools in terms of their high posttraining computational speed and their flexibility to construct complex nonlinear mappings from relatively few known data samples, a survey of past applications of ANNs to the area of core parameter prediction reveals drawbacks such as low prediction accuracy, lack of robust generalization, large network dimensionality, and typically high training requirements. This study provides a brief survey of past and recent applications of ANNs to direct core parameter predictions as well as an alternate hybrid approach that avoids the aforementioned shortcomings of ANNs by combining the mathematical rigor of generalized perturbation theory along with the strong qualities of ANNs in error prediction situations. The results presented focus exclusively on the neutron diffusion’s fundamental mode eigenvalue (i.e., 1/keff) and demonstrate the viability of computationally inexpensive adaptive ANN error controllers for perturbation theory applications.