Data Reconciliation Problem Research Articles

A mixed integer optimization approach for simultaneous data reconciliation and identification of measurement bias

Abstract The problem of data reconciliation and the detection and identification of gross errors, such as measurement bias, are closely related. This close relationship prompted the development of a technique that combines these ideas within a mixed integer optimization framework. This paper describes such an approach and demonstrates its performance with a challenging test problem.

Development of data reconciliation for dynamic nonlinear system: application the polymerization reactor

This work studies the problem of dynamic data reconciliation through a nonlinear dynamic data reconciliation (NLDDR) code based on the dynamic optimization problem with nonlinear constraints associated to a certain calculation horizon. The algorithm is tested on a continuous stirred tank reactor (CSTR) polymerization reactor. A simultaneous strategy of solution and optimization is used to solve the optimization problem. For such a purpose, the optimization problem is turned into a nonlinear programming problem (NLP) through transformation of the ordinary differential equations of the model into a system of algebraic residual equations, which are introduced as a constraint on NLP. The transformation is made using the orthogonal collocation method on finite elements. Successive quadratic programming (SQP) is the technique used to solve the NLP problem, allowing insertion of equalities and inequalities algebraic constraints calculated on-line. To reach a good performance, same methods demand a higher number of samples, which increases the optimization problem dimension and turns the on-line numeric solution to nonviable. An appropriate technique manner to solve or attenuate such a problem is the implementation of calculation horizon along the operation time. Results emphasize the effects of such horizon in the methodology employed.

A Mixed Integer Optimization Approach for Simultaneous Data Reconciliation and Identification of Measurement Bias

The problem of data reconciliation and the detection and identification of gross errors, such as measurement bias, are closely related. This close relationship prompted the development of a technique that combines these ideas within a mixed integer optimization framework. This paper describes such an approach and demonstrates its performance with a challenging test problem.

Regularization of Dynamic Data Reconciliation Problems by Projection 1

Dynamic data reconciliation problems are discussed from the perspective of the mathematical theory of ill-posed inverse problems. Regularization is of crucial importance to obtain satisfactory estimation quality of the reconciled variables. Usually, some penalty is added to the least-squares objective to achieve well-posedness of the problem. However, appropriate discretization schemes of the time-continuous problem act as regularization themselves reducing the need of problem modification. Based on this property, we suggest to successively refine the discretization of the continuous problem starting from a coarse grid to find a suitable regularization which renders a good compromise between (measurement) data and regularization error of the estimate. We conjecture further, that non-equidistant discretization grids are more advantageous than uniform grids.

Industrial Application of a Large-Scale Dynamic Data Reconciliation Strategy

In a modern chemical plant, distributed control systems provide access to a wide variety of process data that typically contain both random and gross errors. Although data reconciliation has been an often-studied topic to reduce the amount of error in measurements, applications of dynamic data reconciliation to problems in industry are virtually nonexistent. This is because standard formulations of the dynamic data reconciliation problem result in large nonlinear programs, which are thought to be too difficult to solve in real time. With increases in computing speeds and improvements in optimization technology, these concerns are diminishing. This paper describes the implementation of a dynamic data reconciliation application at an ExxonMobil Chemical Company plant. The specifics of the application will be discussed and some solutions to the practical problems encountered in an industrial setting are presented.

Simultaneous state and parameter estimation. Application to rainfall data validation

For process control improvement, coherency of information supplied by sensors must first be ensured. Because of the presence of random and possibly gross errors, the model equations of the process are not generally satisfied. The problem of data reconciliation in order to satisfy the model constraints is considered in this article. The simultaneous presence of errors in process input and output measurements poses serious problem in the rectification of data. The proposed procedure to solve this problem involves the use of a special filter to estimate both the parameters, the states and the inputs of a process; smoothing of both estimations of the input and the output is increased by adding in the filter a variance term. Application is proposed for rainfall data validation in order to improve urban sewer network control.

Reconciliation of process data using other projection matrices

Two other projection matrices, used in the solution of data reconciliation problems, are described in this short note. The first matrix projection introduced is straightforward to compute, is idempotent and can be easily updated when a measurement is deleted or removed from the problem. The second projection matrix, while although being more numerically intensive in its computation than the first, may prove superior when ill-behaved or ill-conditioned systems are reconciled given that it employs the very numerically stable singular value decomposition. Two small examples, one non-linear and the other linear, are presented which demonstrate the use of the new projection matrices and serve as a comparison to the well-known matrix projection of Crowe, Garcia, & Hrymak (1983).

Data reconciliation of an agitated thin film evaporator using Aspenplus

Process measurements made in chemical plants often do not satisfy material and energy balances due to random or possibly gross errors in the measuring device readings. Data reconciliation is a method of adjusting process measurements in a weighted least squares sense in order to satisfy the process constraints as well as minimise the difference between the process variable and the measurement of the process variable. Steady state simulation packages equipped with optimisation routines can be used effectively to perform data reconciliation and parameter estimation using existing built-in models which automatically satisfy mass and energy balance constraints. The time required to develop a data reconciliation problem can be significantly reduced by using these packages without sacrificing the quality of the results. Data reconciliation was performed on data from a pilot plant Agitated Thin Film Evaporator (ATFE) for concentrating sugar syrup using the AspenPlus simulation system. An AspenPlus simulation of the ATFE was developed using the built-in rigorous heat exchanger model, Heatx, and the rigorous two phase flash model, Flash2. The optimisation was carried out using the built-in SQP, (Sequential Quadratic Programme), optimisation routine. Data reconciliation improved the fit between the experimental data and the simulation model from between 30% and 98% as well as guaranteeing that the adjusted process measurements satisfy the mass and energy balances. The time required to develop the AspenPlus simulation and data reconciliation was about 1 h whereas a first principles simulation model developed using the programming language such as Fortran or C would take much longer to develop.

On finding the matrix projection in the data reconciliation solution

The focus of this note is to highlight two relatively simple approaches to determine the matrix projection first introduced by Crowe et al. (1983) to solve data reconciliation problems when unmeasured variables exist. The first method uses recursive matrix inversion by partition where the second uses a modified Cholesky factorization. The purpose of the two algorithms is to identify dependent columns and rows in the topology matrix of the unmeasured variables where the matrix projection is then formulated. Although other methods are available to determine the nullity of a matrix such as QR factorization and singular value decomposition it is preferred to use this identification procedure along with Crowe’s matrix projection formulation because of its numerical efficiency simplicity and interpretation. Two mass reconciliation examples are presented one small and one large to clarify and verify the techniques.

The numerical solution of bilinear data reconciliation problems using unconstrained optimization methods

A new approach to solving steady-state data reconciliation problems with bilinear constraints is proposed. Previously it was shown by Crowe that these problems can be solved using the method of matrix projection. In this work the objective function and its constraints are put into unconstrained form using Lagrange multipliers. Unconstrained optimization methods based on analytical derivatives are then used to solve the unconstrained formulation. The unconstrained approach is tested on two different reconciliation problems from the literature using number of Newton-type unconstrained optimization methods. The unconstrained methods are compared in terms of robustness and efficiency in solving the example problems. The performance of the unconstrained approach is compared to that of the method of matrix projection and the projected Lagrangian algorithm implemented in GAMS/MINOS.

A strategy for simultaneous dynamic data reconciliation and outlier detection

The presence of outliers corrupts the procedure of dynamic data reconciliation. In this note, a cluster analysis technique is suggested as a way for discriminating outliers and normal observation data. Furthermore, the formulation of the dynamic data reconciliation problem is modified to incorporate the outlier information. In this way, dynamic reconciliation can be carried out simultaneously with outlier detection. The performance of the proposed approach is demonstrated by simulations on a chemical engineering example from literature.

Use of orthogonal transformations in data classification-reconciliation

In this paper, the use of orthogonal factorizations, more precisely the Q- R decomposition, to analyze, decompose and solve the linear and bilinear data reconciliation problem is further investigated. It is shown that the decomposition provides additional insight in identifying structural singularities in the system topology, allowing the problem to decompose into lower dimension subproblems. Energy balances are explicitly considered. Two examples of application are presented.

The numerical solution of bilinear data reconciliation problems using unconstrained optimization methods

A new approach to solving steady state data reconciliation problems with bilinear constraints is proposed. Previously it was shown by Crowe that these problems can be solved using the method of matrix projection. In this work the objective function and its constraints are put into unconstrained form using Lagrange multipliers. Unconstrained optimization methods based on analytical derivatives are then used to solve the unconstrained formulation. The unconstrained approach is tested on two different reconciliation problems from the literature using a number of Newton-type unconstrained optimization methods. The unconstrained methods are compared in terms of robustness and efficiency in solving the example problems. The performance of the unconstrained approach is compared to that of the method of matrix projection and the projected Lagrangian algorithm implemented in GAMS/MINOS.

Data Reconciliation Using AspenPlus

AbstractProcess measurements made in chemical plants generally do not satisfy material and energy balance constraints due to random or possibly gross errors in the measuring device readings. Data reconciliation is a method of adjusting random errors in the measurements in a weighted least squares sense in order to satisfy the process constraints. Linear data reconciliation involves solving a linear system of mass and energy balances as well as inequality constraints. Non‐linear data reconciliation, involving non‐linear mass and energy balances and constraints, is substantially more complex and requires a significant amount of work in developing the model and/or solution strategy.Steady state simulation packages equipped with optimization routines can be used to perform data reconciliation and parameter estimation with existing models which automatically satisfy mass and energy balance constraints. The time required to develop a data reconciliation problem can be shortened by using these packages without sacrificing the quality of the results. Five examples are presented to illustrate this technique using the AspenPlus flowsheet simulation and Optimization system. The first four are simple problems taken from literature and are included to validate the method. The fifth example involves a nonlinear industrial distillation example included to illustrate the scope of the technique.In the fist four cases, the solution to the data reconciliation problem war easily developed and quickly solved. The fifth example involved somewhat more work but was still relatively quickly developed and solved. Steady state process simulation systems can be used to perform non‐linear data reconciliation. The reduction in development time increases dramatically with the problem complexity involving staged separation and multiple units.

An industrial application of an on-line data reconciliation and optimization problem

A three years project on the installation in an Olefins Plant of an on-line Optimizer is described in some details for the following items: Plant Modeling, Plant Constraints, Plant Measures, Data Reconciliation, On-line Optimization, Numerical Approaches and Results. The conclusions drawn by the paper indicate that Plant Optimization is today the main advisory tool for those Plant Operation Managers who aim to reduce costs and increase Plant profit while operating the plant in safe conditions.

Comparison of Techniques for Data Reconciliation of Multicomponent Processes

Data reconciliation techniques are beginning to be widely applied in many different chemical and mineral beneficiation processes. However, a majority of the applications are for either single columns or a sequence of separation units. Efficient techniques have been proposed to solve data reconciliation problems of such bilinear processes. In this paper we compare two such methods (Crowe, C. M. AIChE J. 1986, 32, 616−623; Simpson, D. E.; Everett, M. G.; Voller, V. R. Appl. Math. Modelling 1988, 12, 204−212) on typical data from multicomponent industrial processes. Although Simpson's method is shown to be more efficient and accurate, the need to generalize it further is pointed out.

An Efficient Formulation for Batch Reactor Data Reconciliation

An efficient formulation of the data reconciliation problem applied to batch reactor experiments is proposed and discussed in comparison with other works reported on this subject in the literature. Under the assumption that all components in the reaction are known experimentally and that their chemical compositions are defined, atomic balances are used as constraints in order to correct raw data values. The formulation is based on an a posteriori simultaneous treatment of all measurements over the whole time of experimentation. The main point of interest in this formulation is the preservation of the measurement identity in each sample. Even if component holdups are systematically involved in mass balances by pairs to estimate a difference in time, here we correct each component measurement and each sample independently. On this basis, it is shown how classical statistical tests can be used to discriminate gross errors. A case study built from simulated true and noisy data gives an overview of the performances of the method. The results state that the overall time formulation is more powerful than the classical one, which is essentially based on a sequential treatment by periods of time. Evaluation of the performance for three adapted statistical tests leads us to conclude that the generalized likelihood ratio (GLR) is not only the most appropriate but also the most efficient test.

Non-linear data reconciliation for an industrial pyrolysis reactor

A comprehensive data reconciliation package for an industrial pyrolysis reactor based on simplified non-linear mass and energy balances was developed. Both successive Linearisation and Successive Quadratic Programming (SQP) methods were used to solve the data reconciliation problem. The package was tested using plant data collected at regular interval over a full generation/regeneration cycle of the reactor. Presence of gross errors in the data set was determined using a global test, and these were detected and rectified using Serial Elimination analysis.

A method to incorporate bounds in data reconciliation and gross error detection—I. The bounded data reconciliation problem

Data reconciliation and gross error detection techniques can be improved by exploiting information from bounds on process variables. In this paper a new approach is proposed for incorporating upper and lower bounds on process variables in data reconciliation and gross error detection. Bounds on process variables are directly incorporated as constraints in data reconciliation and the resulting problem is solved using an efficient quadratic programming algorithm. More importantly, a method to obtain the statistical distributions of measurement residuals and constraint residuals has been developed which is useful for gross error detection. Gross error detection methods based on this approach are described in Part II of this series ( Comput. chem. Engng 17, 1121–1128).

A method to incorporate bounds in data reconciliation and gross error detection—II. Gross error detection strategies

In Part I ( Computers chem. Engng 17, 1115–1120), a procedure was described for solving the data reconciliation problem which includes bounds on process variables. The statistical distributions of the constraint and measurement residuals were also derived. In this paper, two methods are proposed for gross error detection that make use of the results of Part I. One of the methods makes use of bound information in gross error detection while the other does not. The sequence in which gross error detection and data reconciliation are performed is also different in the two methods. Simulation results show that compared to currently available methods, the proposed methods give better gross error detection performance and more accurate estimates which always satisfy the bounds especially when tight bounds are specified.