Output Weights Research Articles

Effluent Quality Soft Sensor for Wastewater Treatment Plant with Ensemble Sparse Learning-Based Online Next Generation Reservoir Computing

Real-time monitoring of key quality variables is essential and crucial for stable and safe operations of wastewater treatment plants (WWTPs). Next generation reservoir computing (NG-RC) has recently garnered significant attention in quality prediction, such as COD and BOD, as an effective alternative to traditional reservoir computing (RC), then is able to act as a data-driven soft sensor to twin a hardware sensor for quality variable measurements. Unlike RC, NG-RC does not require random sampling matrices to define the weights of recurrent neural networks and has fewer hyperparameters. However, NG-RC is usually used online but trained offline, thus leading to model degradation under dynamic scenarios. This paper proposes a sparse online NG-RC approach to meet the real-time requirements of WWTPs and mitigate the impact of measurement noise on the model. First, inspired by the Woodbury matrix identity, an incremental strategy is designed, using sequentially arriving data blocks to learn the output weights of NG-RC online. Then, an ensemble sparse strategy is combined to alleviate overfitting issues of the prediction model. Moreover, a soft sensor based on the ensemble sparse online NG-RC is developed to perform real-time prediction of quality indicators in wastewater treatment processes. Finally, two datasets from actual WWTPs are used to validate the effectiveness of the proposed model.

Enhanced predictive PDF control of stochastic distribution systems with neural network compensation and its application

Compared to the conventional control algorithms that use mean and variance as indicators, predictive probability density function (PDF) control can effectively handle the output PDF control problem of non-Gaussian stochastic distribution systems. However, the existing predictive PDF control method does not consider the construction error between output PDF and weight, thus the control performance is still unsatisfactory. Therefore, this paper proposes a new Enhanced Predictive PDF control method (En-PDF) to improve the output PDF control performance of stochastic distribution systems. The proposed method mainly consists of two parts: the predictive PDF control part and the neural network compensation control part aiming to reduce the bias of output PDF. First, the Radical Basis Functions (RBFs) are used to approximate the PDF of the stochastic systems output, and then a prediction model representing the relationship between input and weight is established using the subspace identification algorithm to design the predictive PDF control for the stochastic systems. Next, the Kullback-Leibler (KL) divergence is used to measure the similarity between the output PDF and the set PDF, combined with the weight error and compensation to design a new performance index. Based on this, the parameters of the neural network are adjusted using the gradient descent algorithm to obtain the optimal compensation, and the stability and tracking performance of the proposed algorithm are analyzed using inductive reasoning method. Finally, the predictive PDF control input with the compensation work together on the controlled plant to achieve high-performance control of the output PDF of non-Gaussian stochastic distribution systems. Both simulation experiments and physical control experiments validate the effectiveness and superiority of the proposed method.

Reservoir structure optimization of echo state networks: A detrended multiple cross-correlation pruning perspective

Reservoir structure optimization of echo state networks (ESN) is an important enabler for improving network performance. In this regard, pruning provides an effective means to optimize reservoir structure by removing redundant components in the network. Existing studies achieve reservoir pruning by removing insignificant neuronal connections. However, such processing causes the optimized neurons to still remain in the reservoir and thus hinder network inference by participating in computations, leading to suboptimal utilization of pruning benefits by the network. To solve this problem, this paper proposes an adaptive pruning algorithm for ESN within the detrended multiple cross-correlation (DMC2) framework, i.e., DMAP. On the whole, it contains two main functional parts: DMC2 measure of reservoir neurons and reservoir pruning. Specifically, the former is used to quantify the correlation among neurons. Based on this, the latter can remove neurons with high correlation from the reservoir completely, and finally obtain the optimal network structure by retraining the output weights. Experiment results show that DMAP-ESN outperforms its competitors in nonlinear approximation capability and reservoir stability.

Hybrid Extreme Learning for Reliable Short-Term Traffic Flow Forecasting

Reliable forecasting of short-term traffic flow is an essential component of modern intelligent transport systems. However, existing methods fail to deal with the non-linear nature of short-term traffic flow, often making the forecasting unreliable. Herein, we propose a reliable short-term traffic flow forecasting method, termed hybrid extreme learning, that effectively learns the non-linear representation of traffic flow, boosting forecasting reliability. This new algorithm probes the non-linear nature of short-term traffic data by exploiting the artificial bee colony that selects the best-implied layer deviation and input weight matrix to enhance the multi-structural information perception capability. It speeds up the forecasting time by calculating the output weight matrix, which guarantees the real usage of the forecasting method, boosting the time reliability. We extensively evaluate the proposed hybrid extreme learning method on well-known short-term traffic flow forecasting datasets. The experimental results show that our method outperforms existing methods by a large margin in both forecasting accuracy and time, effectively demonstrating the reliability improvement of the proposed method. This reliable method may open the avenue of deep learning techniques in short-term traffic flow forecasting in real scenarios.

On value efficiency analysis and cone-ratio data envelopment analysis models

AbstractIn this paper we relate the Value Efficiency Analysis (VEA) efficiency scores with those of three different Cone-Ratio Data Envelopment Analysis (CR-DEA) models, when the chosen model Decision Making Units (DMUs) in CR-DEA jointly comprise the Most Preferred Solution (MPS) in VEA. In particular, we consider the cases where the set of feasible weight vectors is given respectively by (i) the weight vectors that are jointly optimal in DEA for all the model DMUs, (ii) every weight vector that is optimal in DEA for at least one of the model DMUs, and (iii) only those weight vectors with strictly positive input and output weights, each of which is jointly optimal in DEA for all the model DMUs. In each of these cases we show that the CR-DEA efficiency scores can be obtained or approximated by estimating either a VEA model or a series of VEA models, without the need of identifying a priori the set of feasible weight vectors. We illustrate the usefulness of our results by means of an empirical application using data on Japanese regional banks.

Evaluating the effectiveness of the TOPSIS approach for three-wire electrode machining of D2 steel using the wire EDM method

Due to the high demand for D2 steel as a tool material and the difficulty of the machine, optimization of process parameters in advanced manufacturing machines is needed. This study investigates three wire electrodes: brass, coated copper, and annealed copper, analyzing their impact on tool material. Employing 0.25 mm wires, 10 mm D2 steel cubes are cut for consistent comparison. An L27 orthogonal array tests six parameters at three levels, optimizing with analytic hierarchy process technique for order preference by similarity to the ideal solution (AHP-TOPSIS). The response parameters were the metal removal rate (MRR) and kerf width (KW). Pulse on/off time, wire tension, spark voltage, input current, and wire feed rate vary systematically for each wire. The tests validate the efficacy of the AHP-TOPSIS method in optimizing WEDM parameters and machining performance. ANOVA reveals pulse-on and pulse-off times as crucial factors for various wire electrodes. Under diverse conditions, pulse duration increases spark efficiency. Based on the AHP-TOPSIS method results, weights of outputs revealed that the annealed copper wire yields the highest MRR value (0.232 mm3/sec). The brass wire exhibited the lowest MRR value (0.127 mm3/sec) compared to the others.

Robust multi-target regression with improved stochastic configuration networks and its applications

To improve the accuracy and robustness of stochastic configuration networks (SCNs) for resolving multi-target regression tasks, this paper proposes a robust modeling approach based on improved stochastic configuration networks. A parallel implementation of SCN models is designed to incrementally generate the hidden nodes, which enhances the diversity of hidden layer mapping through information superposition and spanning connection. We employ an elastic net regularization model to sparsely constrain the model parameters to characterize the correlation among multiple targets. Then, the mixture Laplace distributions are used as the prior distribution of each target modeling error, and the output weights of the SCN model are re-evaluated by maximizing a posteriori estimation to enhance model’s robustness with respect to some uncertainties presented in training samples. The modelling performance of the proposed solution is tested on six standard datasets and the historical data of a municipal solid waste incineration process. The experimental results show that the proposed modeling technique has advantages in terms of both the prediction accuracy and the robustness.

Modular stochastic configuration network with attention mechanism for soft measurement of water quality parameters in wastewater treatment processes

Due to the complex and multi-distributed nature of wastewater treatment process data, stochastic configuration networks (SCNs) face difficulties such as large network structure and complex construction process when dealing with data modeling tasks. To address these challenges, this paper proposes a modular stochastic configuration network with attention mechanism, which integrates the advantages of modular partitioning, attention mechanism, and SCN. The model employs a fuzzy mean clustering strategy to simplify and decompose the complex multi-case and multi-distribution nonlinear modeling task, thereby reducing the modeling complexity of a single model. Furthermore, the model utilizes a stochastic configuration algorithm to learn the subtasks and construct corresponding sub-networks. The output weights are then assigned to each sub-network based on the attention mechanism to obtain the model’s final output. The validity of the proposed method is validated through two benchmark experiments. Then the model is applied to measure effluent ammonia nitrogen. The results indicate that compared with the classical randomized and modularized network models, the proposed model exhibits certain advantages in terms of learning accuracy and generalization performance.

Modal-Regression-Based Broad Learning System for Robust Regression and Classification.

A novel neural network, namely, broad learning system (BLS), has shown impressive performance on various regression and classification tasks. Nevertheless, most BLS models may suffer serious performance degradation for contaminated data, since they are derived under the least-squares criterion which is sensitive to noise and outliers. To enhance the model robustness, in this article we proposed a modal-regression-based BLS (MRBLS) to tackle the regression and classification tasks of data corrupted by noise and outliers. Specifically, modal regression is adopted to train the output weights instead of the minimum mean square error (MMSE) criterion. Moreover, the l2,1 -norm-induced constraint is used to encourage row sparsity of the connection weight matrix and achieve feature selection. To effectively and efficiently train the network, the half-quadratic theory is used to optimize MRBLS. The validity and robustness of the proposed method are verified on various regression and classification datasets. The experimental results demonstrate that the proposed MRBLS achieves better performance than the existing state-of-the-art BLS methods in terms of both accuracy and robustness.

Learning to Guide Particle Search for Dynamic Multiobjective Optimization.

Dynamic multiobjective optimization problems (DMOPs) are characterized by multiple objectives that change over time in varying environments. More specifically, environmental changes can be described as various dynamics. However, it is difficult for existing dynamic multiobjective algorithms (DMOAs) to handle DMOPs due to their inability to learn in different environments to guide the search. Besides, solving DMOPs is typically an online task, requiring low computational cost of a DMOA. To address the above challenges, we propose a particle search guidance network (PSGN), capable of directing individuals' search actions, including learning target selection and acceleration coefficient control. PSGN can learn the actions that should be taken in each environment through rewarding or punishing the network by reinforcement learning. Thus, PSGN is capable of tackling DMOPs of various dynamics. Additionally, we efficiently adjust PSGN hidden nodes and update the output weights in an incremental learning way, enabling PSGN to direct particle search at a low computational cost. We compare the proposed PSGN with seven state-of-the-art algorithms, and the excellent performance of PSGN verifies that it can handle DMOPs of various dynamics in a computationally very efficient way.

Generalized [formula omitted]-sparse algorithms for constructing fault tolerant RBF networks

In the construction process of radial basis function (RBF) networks, two common crucial issues arise: the selection of RBF centers and the effective utilization of the given source without encountering the overfitting problem. Another important issue is the fault tolerant capability. That is, when noise or faults exist in a trained network, it is crucial that the network’s performance does not undergo significant deterioration or decrease. However, without employing a fault tolerant procedure, a trained RBF network may exhibit significantly poor performance. Unfortunately, most existing algorithms are unable to simultaneously address all of the aforementioned issues. This paper proposes fault tolerant training algorithms that can simultaneously select RBF nodes and train RBF output weights. Additionally, our algorithms can directly control the number of RBF nodes in an explicit manner, eliminating the need for a time-consuming procedure to tune the regularization parameter and achieve the target RBF network size. Based on simulation results, our algorithms demonstrate improved test set performance when more RBF nodes are used, effectively utilizing the given source without encountering the overfitting problem. This paper first defines a fault tolerant objective function, which includes a term to suppress the effects of weight faults and weight noise. This term also prevents the issue of overfitting, resulting in better test set performance when more RBF nodes are utilized. With the defined objective function, the training process is designed to solve a generalized M-sparse problem by incorporating an ℓ0-norm constraint. The ℓ0-norm constraint allows us to directly and explicitly control the number of RBF nodes. To address the generalized M-sparse problem, we introduce the noise-resistant iterative hard thresholding (NR-IHT) algorithm. The convergence properties of the NR-IHT algorithm are subsequently discussed theoretically. To further enhance performance, we incorporate the momentum concept into the NR-IHT algorithm, referring to the modified version as “NR-IHT-Mom”. Simulation results show that both the NR-IHT algorithm and the NR-IHT-Mom algorithm outperform several state-of-the-art comparison algorithms.

Determining the equilibrium efficient frontier by proportional frontier shifting for data envelopment analysis with fixed-sum outputs

The equilibrium efficient frontier data envelopment analysis (EEFDEA) has been extensively used to evaluate efficiencies of the decision-making units (DMUs) with fixed-sum outputs. This study develops a new EEFDEA approach based on a proportional frontier-shifting strategy. Our approach applies an iterative procedure to find the equilibrium efficient frontier (EEF). Each iteration uses a proportional frontier-shifting model to improve an inefficient DMU to the efficient frontier by increasing its fixed-sum outputs. Meanwhile, the DMUs on the efficient frontier decrease fixed-sum outputs proportionally to ensure the total fixed-sum outputs are unchanged. Our theoretical developments show that the proportional frontier-shifting strategy is feasible and can finally obtain a unique EEF. The new approach allows DMUs to use their preferred input and output weights when determining the EEF. This generates an EEF that better aligns with real-world practices and avoids the need to construct it as a single hyperplane, as required by conventional EEFDEA methods. It also avoids unfair adjustments in fixed-sum outputs among the DMUs and eliminates the problem of peculiar efficiency evaluation results (i.e., some DMUs obtain extremely high, or infinity, efficiencies). Finally, we apply our approach to a case study of Chinese vehicle industry companies to demonstrate its usefulness and compare it with the previous representative approach.

Efficiency evaluation of 28 health systems by MCDA and DEA

BackgroundPolicymakers, who are constantly discussing growing health expenditures, should know whether the health system is efficient. We can provide them with such information through international health system efficiency evaluations. The main objectives of this study are: (a) to evaluate the efficiency of health systems in 28 developed countries by multiple-criteria decision analysis (MCDA) and data envelopment analysis (DEA) and (b) to identify reasonable benchmark countries for the Czech Republic, for which we collect information on the relative importance of health system inputs and outputs.MethodsWe used MCDA and DEA to evaluate the efficiency of the health systems of 28 developed countries. The models included four health system inputs (health expenditure as a relative share of GDP, the number of physicians, nurses, and hospital beds) and three health system outputs (life expectancy at birth, healthy life expectancy, and infant mortality rate). The sample covers 27 OECD countries and Russia, which is also included in the OECD database. To determine the input and output weights, we used a questionnaire sent to health policy experts in the Czech Republic.ResultsWe obtained subjective information on the relative importance of the health system inputs and outputs from 27 Czech health policy experts. We evaluated health system efficiency using four MCDA and two DEA models. According to the MCDA models, Turkey, Poland, and Israel were found to have efficient health systems. The Czech Republic ranked 16th, 19th, 15th, and 17th. The benchmark countries for the Czech Republic’s health system were Israel, Estonia, Luxembourg, Italy, the UK, Spain, Slovenia, and Canada. The DEA model with the constant returns to scale identified four technically efficient health systems: Turkey, the UK, Canada, and Sweden. The Czech Republic was found to be one of the worst-performing health systems. The DEA model with the variable returns to scale identified 15 technically efficient health systems. We found that efficiency results are quite robust. With two exceptions, the Spearman rank correlations between each pair of models were statistically significant at the 0.05 level.ConclusionsDuring the model formulation, we investigated the pitfalls of efficiency measurement in health care and used several practical solutions. We consider MCDA and DEA, above all, as exploratory methods, not methods providing definitive answers.

A Recurrent Stochastic Configuration Network for Dynamic System Modeling and Its Application

In order to model nonlinear dynamic systems quickly and accurately, a recurrent stochastic configuration network (RSCN) with universal approximation ability is proposed in this paper. The method consists of network parameter learning and network structure determination. The network parameter learning strategy consists of input weights and the biases of hidden nodes generated randomly through a supervision mechanism, while the output weights of hidden nodes are determined by solving the least-squares problem. The network structure is determined incrementally by the training error of a single-layer recurrent neural network. The effectiveness of RSCN is tested and evaluated by using three nonlinear dynamic functions and historical data from a municipal solid waste incineration (MSWI) process. The experimental results show that the proposed method can improve modeling accuracy and reduce training time, which is valuable for applications in the field of industrial process modeling.

Safe dynamic sparse training of modified RBF networks for joint feature selection and classification

Developing an efficient embedded feature selection method for both binary and multiclass classification problems is a fundamental topic to be further studied. In this paper, the conventional radial basis function (RBF) network is modified for joint feature selection and classification. By utilizing the anisotropic Gaussian basis function, an individual weight parameter is assigned to each feature for feature weighting. The L1-regularized loss function is designed by combining the softmax cross entropy loss with L1 regularization terms imposing on feature and output weights. A specialized active-set limited-memory projected Quasi-Newton (SAL-PQN) algorithm is proposed to minimize the L1-regularized loss function, which can optimize all the adjustable parameters and zero out the redundant feature and output weights simultaneously. Furthermore, a safe dynamic pruning strategy is integrated into the SAL-PQN algorithm for dynamic sparse training of the modified RBF network, which continuously excludes the dynamically zeroed-out feature weights from the training process. It is theoretically assured of yielding the same gradient updates as SAL-PQN, thus the relevant computation of the zeroed-out feature weights in the forward and backward propagations can be safely eliminated. The experimental results demonstrate the effectiveness and efficiency of the proposed methods for joint feature selection and classification, and illustrate the practical utility of the inherent feature weighting capability.

Greedy deep stochastic configuration networks ensemble with boosting negative correlation learning

Deep stochastic configuration networks (DSCNs) employ data-dependent supervision mechanism to randomly assign node parameters and incrementally construct the deep neural network structure, thereby ensuring the model's universal approximation property. To build a random neural networks ensemble model with better generalization performance, we propose a novel greedy deep stochastic configuration networks ensemble model based on boosting negative correlation learning, termed as GDSCNE. Firstly, greedy optimization strategy based on inequality constraints is utilized to generate random parameters of base components with multi-layer architecture, which can accelerate the decline of network residuals when configuring a new node. Additionally, boosting negative correlation learning framework is presented for the base components ensemble process, which uses least square approach with negative correlation learning penalty term to update the ensemble output weights for each base component, subsequently, boosting method is applied to construct a stronger ensemble model by adaptive weighting through the results of base components. Finally, we evaluated GDSCNE on the popular regression benchmark datasets from the KEEL, experimental results demonstrate that GDSCNE outperforms state-of-the-art random learning algorithms in terms of regression accuracy and generalization performance across several regression datasets with varying sizes.

ConGCNet: Convex geometric constructive neural network for Industrial Internet of Things

The intersection of the Industrial Internet of Things (IIoT) and artificial intelligence (AI) has garnered ever-increasing attention and research interest. Nevertheless, the dilemma between the strict resource-constrained nature of IIoT devices and the extensive resource demands of AI has not yet been fully addressed with a comprehensive solution. Taking advantage of the lightweight constructive neural network (LightGCNet) in developing fast learner models for IIoT, a convex geometric constructive neural network with a low-complexity control strategy, namely, ConGCNet, is proposed in this article via convex optimization and matrix theory, which enhances the convergence rate and reduces the computational consumption in comparison with LightGCNet. Firstly, a low-complexity control strategy is proposed to reduce the computational consumption during the hidden parameters training process. Secondly, a novel output weights evaluated method based on convex optimization is proposed to guarantee the convergence rate. Finally, the universal approximation property of ConGCNet is proved by the low-complexity control strategy and convex output weights evaluated method. Simulation results, including four benchmark datasets and the real-world ore grinding process, demonstrate that ConGCNet effectively reduces computational consumption in the modelling process and improves the model’s convergence rate.

Efficiency optimization methods for stochastic configuration networks

To solve the problems of the low efficiency of parameter allocation and time-consuming computation of output weights in the hidden layers of stochastic configuration networks (SCNs), an optimization method is proposed to improve the SCNs construction efficiency. Firstly, with the increase in the number of hidden layer nodes, the key parameters that determine the strictness of the supervision mechanism are reconstructed to speed up the configuration efficiency of hidden layer input weights and biases. Then, the incremental mechanism of the SCNs are combined with the QR decomposition method, and the output weights are calculated by iteratively updating the transformation matrix, thus reducing the computational complexity of training the SCNs. Finally, the proposed method is validated on four standard datasets and historical data of a municipal solid waste incineration process. The experimental results show that the proposed method improves the efficiency of SCN construction while guaranteeing the prediction accuracy of SCNs model.

Generalized sparse and outlier-robust broad learning systems for multi-dimensional output problems

Broad learning systems (BLSs) are becoming increasingly popular due to their fast and superior learning capabilities. However, their performances are susceptible to outliers and dense networks. The Elastic-net robust BLS (ER-BLS), which employs an ℓ1-norm loss function along with Elastic-net regularization, attempts to mitigate these issues by promoting network sparsity. Nevertheless, even ER-BLS fails to achieve satisfactory robustness and sparsity in multi-dimensional output tasks because the ℓ1-norm ignores the correlations between the output dimensions and output weights. Therefore, we propose a generalized Elastic-net robust BLS (GER-BLS) by generalizing the ℓ1-norm to the L2,1-norm. The L2,1-norm enhances robustness by treating the residuals across all output dimensions as a whole and induces node-level sparsity by penalizing all output weights connected to a single hidden node. The alternating direction method of multipliers (ADMM) algorithm was employed to solve GER-BLS efficiently, circumventing the non-differentiable issue of the objective function. Moreover, we developed DGER-BLS, a distributed variant of GER-BLS, tailored to handle data with distributed storage. The convergence analysis of our proposed algorithms and a large number of numerical experiments on multi-dimensional output datasets demonstrated the effectiveness of GER-BLS and DGER-BLS in dealing with outliers, distributed stored data, and sparsity.

An improved equilibrium efficient frontier data envelopment analysis approach for evaluating decision-making units with fixed-sum outputs

In recent years, significant development has been made in efficiency evaluation for Decision-Making Units (DMUs) with fixed-sum outputs. The Generalized Equilibrium Efficient Frontier Data Envelopment Analysis (GEEFDEA) approach introduced by Yang et al. (2015) is one of the most representative methods. In the GEEFDEA approach, all DMUs are adjusted to become efficient under the same set of weights, indicating that the Equilibrium Efficient Frontier (EEF) constructed consists of only one hyperplane. However, in practical scenarios under the Variable Return to Scale (VRS) assumption, the production frontier always consists of multiple hyperplanes, presenting a more complex shape. To fill this research gap, we propose an improved Equilibrium Efficient Frontier Data Envelopment Analysis approach. Our approach allows DMUs to have different weights for inputs, variable-sum outputs, and fixed-sum outputs, resulting in an EEF with multiple hyperplanes. It is noted that our new approach uses a non-linear model to obtain the EEF. We show that the model can be linearized in the case of a single fixed-sum output. In situations involving multiple fixed-sum outputs, we propose an algorithm based on the Expectation Maximization (EM) mechanism to solve the model to obtain an EEF. Finally, we illustrate the advantages of our new approach through a numerical example and a case study in the global motor vehicle industry.