Data Validation And Reconciliation Research Articles

LSTM-based autoencoder models for real-time quality control of wastewater treatment sensor data

Abstract The operation of smart wastewater treatment plants (WWTPs) is increasingly paramount in improving effluent quality, facilitating resource recovery and reducing carbon emissions. To achieve these objectives, sensors, monitoring systems, and artificial intelligence (AI)-based models are increasingly being developed and utilised for decision support and advanced control. Key to the adoption of advanced data-driven control of WWTPs is real-time data validation and reconciliation (DVR), especially for sensor data. This research demonstrates and evaluates real-time AI-based data quality control methods, i.e. long short-term memory (LSTM) autoencoder (AE) models, to reconcile faulty sensor signals in WWTPs as compared to autoregressive integrated moving average (ARIMA) models. The DVR procedure is aimed at anomalies resulting from data acquisition issues and sensor faults. Anomaly detection precedes the reconciliation procedure using models that capture short-time dynamics (SD) and (relatively) long-time dynamics (LD). Real data from an operational WWTP are used to test the DVR procedure. To address the reconciliation of prolonged anomalies, the SD is aggregated with an LD model by exponential weighting. For reconciling single-point anomalies, both ARIMA and LSTM AEs showed high accuracy, while the accuracy of reconciliation regresses quickly with increasing forecasting horizons for prolonged anomalous events.

An Estimation Method for Bias Error of Measurements by Utilizing Process Data, An Incidence Matrix and a Reference Instrument for Data Validation and Reconciliation

Abstract With further applications of AI, IoT, and digital twin technology to plant operation and maintenance, it is becoming increasingly important to ensure data reliability. Data validation and reconciliation (DVR) represents one promising technique to ensure data reliability by minimizing the uncertainty of measurements based on statistics. DVR has been widely applied to nuclear power electrical generation plants in Europe and the U.S. in recent years. The most important input for DVR analysis is measurement uncertainty. In Japan, performance management of nuclear power plants is often done by measuring condensate flowrate. While the uncertainty of other flowmeters is handled by the JIS standard, the condensate flowmeter is specially calibrated every few cycles. This leads to reduction of effectiveness of DVR analysis due to variations in measurement uncertainty management. To overcome this issue, we propose an estimation method for measurement uncertainty by utilizing process data, an incidence matrix between sensors, and a reference instrument. The conventional method proposed in the previous study only treats the random error. The proposed method quantitatively estimates not only random error but also bias error by considering the uncertainty of the reference instrument. Using several benchmark problems, we found that the proposed method was applicable to various flow conditions, including physically fluctuating flow such as that observed in the feedwater flow in nuclear power plants. We anticipate that the proposed method will promote use of DVR analysis in nuclear power plants in Japan.

Geometrical interpretation of Data Reconciliation and uncertainty reduction in Multi-Sensor systems

A data reconciliation and uncertainty reduction process, for multi-sensor systems such as power production plants, chemical processing plants, and many others, is considered. Methods of correcting original measurements to reduce their uncertainty, and to obtain more reliable information about the system, known as Data Validation and Reconciliation (DVR) or Process Data Reconciliation (PDR), have been developed in recent decades. These methods are present in literature as discussions on computer codes, statistical criteria, and details of specific industrial applications and implementations. This paper provides a mathematically accurate and intuitively understandable discussion on the scientific basis, benefits, limitations, and properties of the DVR/PDR technology. This is achieved through geometrical interpretation of this technology in a multidimensional metric space with metric tensor defined by the set of the original measurements. The effect of various factors on the methods’ output, are analyzed using graphical illustrations and analytical calculations.

Data validation and reconciliation for error correction and gross error detection in multiphase allocation systems

In upstream oil and gas production, flow measurement errors are common at multiple locations in the production system. These measurement errors lead to imbalances, which are reconciled by using allocation methods. The allocation method should redistribute the imbalance in a fair manner to all involved stakeholders. In this paper, data validation and reconciliation (DVR) is proposed as an alternative allocation method in multiphase production systems. DVR is a model-based optimization method that exploits redundancies in process data to minimize random measurement errors, and simultaneously provides a basis for the detection of gross errors. To study the applicability of DVR compared to conventional allocation methods, a model for generic multiphase production systems is developed, and the corresponding DVR problem is formulated. In order to deal with nonlinear flow effects (e.g. interphase mass transfer) the model is linearized around the operating conditions and the DVR problem is solved iteratively. From the simulation studies, it is concluded that DVR reduces measurement errors by up to 56% for gas allocation systems and 33% for multiphase systems. Gross errors in interior locations of the network can be located precisely, while allocating gross errors to exterior locations is generally not possible, due to limited network redundancy. Compared to conventional methods, DVR provides a better means for the detection of gross errors, and results in more accurate attribution of oil/gas ownership and more fair division of revenues between stakeholders.

Methodology of advanced data validation and reconciliation application in industrial thermal processes

Data validation and reconciliation is a technique that uses process information and mathematical methods in order to correct measurements in thermal industrial processes. On the need to measurements data validation and reconciliation (DVR) in order to use them for further calculations is indicated in the Association of German Engineers guidelines – VDI 2048. Moreover, the necessity of removing measurement errors, in the international standard for the integration of enterprise and control systems ISA-95 is recommended. The paper presents in a comprehensive manner methodology of advanced DVR application in computer systems supporting supervising of the industrial thermal processes. The necessity of preliminary measurement data validation has been indicated. The problems of detection and identification of measurements with a gross error were presented. A method that enables the detection of steady and unsteady states in thermal processes has been shown. The statistical tests for control the assumed accuracy of measurements after advanced DVR calculations have been displayed. A comprehensive analysis of the usefulness of these statistical tests used in the detection and identification of gross measurement errors in the advanced DVR method was carried out. The presented computational example illustrating the use of statistical methods for the detection and identification of gross measurement errors clearly shows that the use of these methods does not solve the problem in the case of advanced DVR tasks with the non-linear form of conditional equations. In this case, the additional methods presented in the article must be used. An overall methodology of applying the generalized advanced DVR method in distributed control systems of thermal processes in the industry was developed. Based on that, a block diagram of the worked-out methodology of the application mentioned DVR method in the industrial thermal processes has been elaborated. The developed overall methodology and aggregated block diagram enable an integrated approach to the generalized advanced DVR method in industrial thermal processes.

An improvement of measurements reliability in thermal processes by application of the advanced data reconciliation method with the use of fuzzy uncertainties of measurements

An improvement of the measurements reliability in thermal processes by the application of the advanced data validation and reconciliation (DVR) methodology strictly depends on their assumed uncertainties. In thermal processes, which are realized in industry, the measurements uncertainties are often determined by the use of all available information concerning possible variability of measurement results. Very often, the determination of measurements uncertainties require professional knowledge, however this approach may lead to imprecise results The paper presents an application of the advanced data reconciliation method with the use of uncertainties of measurements expressed by fuzzy numbers. The proposed methodology has been presented based on example of gas-and-steam CHP unit. In order to compare calculation results for deterministic and the fuzzy uncertainties, the complex standard uncertainties of main assessment indicators of analyzed CHP unit operation supervision as well as the relative entropy of information – Kullback-Leibler divergence for two multivariate normal distributions have been chosen. By the use of mentioned criterions, it has been observed that better results are obtained with the proposed method using fuzzy measurements uncertainties. The results of this study showed that calculated values of uncertainties of energy assessment indicators and the relative information entropy of a whole measurements system are both smaller and more reliable.

Protection of a Nuclear Reactor Monitoring System against Gross Measurement Errors

Data Validation and Reconciliation (DVR) is a basic tool for protection of industrial monitoring systems against gross measurement errors. However, the power of this approach is frequently not fully exploited. This paper concentrates on protection of key results against gross and systematic errors. On‐line monitoring can allow a continuous protection of measurement targets against gross errors in measurement chains. The measurement target can be, for example, a nuclear reactor heat output while errors can be hidden in measured flows and state variables of steam and water. The methodology presented here allows a gross error to be identified while maintaining an accurate value of the target variable. In analogy with statistics (power of statistical tests) we can define the probability of success as a power of the Monitoring System Self‐Protection (MSSP). A new simple method for assessing the MSSP power is presented. The MSSP is important for monitoring system maintenance and the optimal placement of additional instruments. It shows which primary variables are self‐checked by the redundancy of the system and which should be checked (calibrated) independently. The problem is illustrated by an example taken from one nuclear power station.

Smart Production Metering Field Pilot: the Benefits from a Virtual Approach

To face new hydrocarbon production metering systems and back allocation challenges, Total E&P Qatar and its Research Center (TRC-Q), located in the Qatar Science and Technology Park (QSTP), test novel approaches to improve current production metering system in the Al-Khalij field. This field is operated under a Production Sharing Agreement (PSA) with Qatar Petroleum (QP). A field pilot, which involves three platforms offshore, is currently conducted to test a Data Validation and Reconciliation (DVR) software. This tool is used for production accounting, monitoring and as an alarm system related to equipments’ failure. The main outcome to be expected from this pilot will be an automatic validation of on-line production data before their final recording in the database. The key advantage of the DVR approach is to take into account all the information redundancy and data uncertainties. In the DVR approach, a realistic uncertainty is associated with the information (measurement / model parameter) and a statistical method is applied to re-estimate and improve simultaneously the information and its subsequent uncertainty. The current status of this study is presented along with improvements and challenges. A few examples of on line monitoring of Al-Khalij Wells’ production are presented through a comparison between the DVR on-line virtual meter outputs and the on-site production tests. The results, obtained from sensitivity analysis are also discussed to assess the improvements achieved by applying the DVR approach.

Up-stream smart metering pilot

Total E&P Qatar seeks to constantly improve its operations performances. These performances are associated with such indicators as health, safety & environment (HSE) performance, oil production, water management, production costs and energy efficiency. A key tool for reaching this goal is the enhancement of data monitoring and management technology. The smart metering pilot project is run at Total Research Centre - Qatar (TRC-Q) at Qatar Science and Technology Park in close cooperation with Total E&P Qatar Operations and TOTAL Research in France. At TRC-Q, we are testing a game-changing technology based on data validation and reconciliation (DVR). We integrate real-time measurements and virtual metering in an online monitoring system on our production platforms. The originality compared to equivalent systems is the use of the DVR approach, which consists of using a statistical model to manage uncertainties associated with each measured parameter and of quantifying error propagation. The DVR process allows an automatic real-time correction of both measurements and model parameters, on the basis of their allocated uncertainty and thanks to the information redundancy. Consequently, the resulting output data is more consistent with the available data and associated with a reduced uncertainty. So, for the DVR to be effective we seek to increase the data redundancy and provide high-quality modelling of physical and thermodynamic phenomena occurring over the whole production process. The achievement so far is a pilot hourly metering system based on online measurements. These measurements and modelling parameters are automatically linked on the basis of their redundancy and the Operations Department is provided with a customised report containing the corrected parameters and their associated uncertainties. We also keep looking for new relevant information to incorporate into the system and for improvement in the quality of the mathematical model associated with each equipment (pumps, flow meters, etc.) model in order to improve the reliability of the output. The results that will be presented cover the initial promising results of the pilot for the virtual metering of oil, water and gas production and the associated uncertainty quantification.

Transforming Production Data Into Knowledge

This article, written by Assistant Technology Editor Karen Bybee, contains highlights of paper SPE 112517, "Transforming E&P Data Into Knowledge: Applications of an Integration Strategy," by Jean-Paul Couput, Alain Louis, SPE, and Jacques Danquigny, SPE, Total S.A., originally prepared for the 2008 SPE Intelligent Energy Conference and Exhibition, Amsterdam, 25-27 February. The paper has not been peer reviewed. Data validation and reconciliation (DVR) applied to flow-measurement systems improves the reliability and accuracy of production estimates. The potential of DVR to analyze, quantify, and interpret large amounts of data and provide more information from existing sensors is of particular value when key measurement devices fail or are not available. Introduction The main objective of the digital-oilfield concept is to maximize production and recovery by integrating all production operations, from the reservoir to export, by use of suitable models and optimization techniques. The combination of both existing and leading-edge technologies ensures that field optimal efficiency is achieved. Today, most production systems require accurate raw production data. All measured data contain various types of errors (e.g., intrinsic measurement errors, gross errors, and noise). Moreover, when real-time measuring systems, such as multiphase flowmeters (MPFMs), are used, errors may originate from the inaccuracy of the model used to compute the flow rates of the individual phases. Operating decisions based on such data can affect reserves calculations. The traditional approach to overcome these issues by combining best practices and measurement devices is not sufficient any longer. In the upstream area, new sensor technologies including three-phase metering and virtual metering have emerged. They still remain a challenge, especially in deepwater subsea applications, where equipment repair or replacement involves large costs. One solution is to combine data from physical sensors with virtual metering and DVR models. This is an innovative approach that has been proved in the downstream area and that effectively helps increase oil production and recovery. Data-Quality Issues Building strong and robust field-monitoring systems assumes that the right and correct measurement and data are made available for further processing at the right time. This constitutes a real challenge for existing and future upstream installations, where both instrumentation and associated measurements are deficient and inconsistent. Another issue is the lack of measurement in case of a failure or in the case of measurement systems designed with minimum equipment. Such questions are not new, but now they are critical and more difficult to address, mainly because requirements are more stringent and measurement accessibility and quality control (QC) are more problematic, especially in deepwater-field developments and in unmanned situations.