Surveillance Error Research Articles

Blood Gas Analyzer Accuracy of Glucose Measurements

ObjectiveTo investigate the comparability of glucose levels measured with blood gas analyzers (BGAs) and by central laboratories (CLs). Material and MethodsGlucose measurements obtained between June 1, 2007, and March 1, 2016, at the Vanderbilt University Medical Center were reviewed. The agreement between CL and BGA results were assessed using Bland-Altman, consensus error grid (CEG), and surveillance error grid (SEG) analyses. We further analyzed the BGAs’ performance against the US Food and Drug Administration (FDA) 2014 draft guidance and 2016 final guidance for blood glucose monitoring and the International Organization for Standardization (ISO) 15197:2013 standard. ResultsWe analyzed 2671 paired glucose measurements, including 50 pairs of hypoglycemic values (1.9%). Bland-Altman analysis yielded a mean bias of −3.1 mg/dL, with 98.1% of paired values meeting the 95% limits of agreement. In the hypoglycemic range, the mean bias was −0.8 mg/dL, with 100% of paired values meeting the 95% limits of agreement. When using CEG analysis, 99.9% of the paired values fell within the no risk zone. Similar results were found using SEG analysis. For the FDA 2014 draft guidance, our data did not meet the target compliance rate. For the FDA 2016 final guidance, our data partially met the target compliance rate. For the ISO standard, our data met the target compliance rate. ConclusionIn this study, the agreement for glucose measurement between common BGAs and CL instruments met the ISO 2013 standard. However, BGA accuracy did not meet the stricter requirements of the FDA 2014 draft guidance or 2016 final guidance. Fortunately, plotting these results on either the CEG or the SEG revealed no results in either the great or extreme clinical risk zones.

Continuous glucose monitoring following pancreatectomy with islet autotransplantation in children.

Aim was to determine whether CGM could accurately monitor blood glucose concentration in the immediate postoperative period following pancreatectomy with IAT in children. CGM was used in nine patients undergoing IAT at our institution between April 2015 and September 2016 (eight total pancreatectomy and one subtotal pancreatectomy). MAD and MARD of CGM values compared to time-matched serum blood glucose were calculated during the first 5days of ICU admission. Goal range was defined as 70-140mg/dL and out-of-range was >140mg/dL or <70mg/dL. Of 89 time-matched measures found, 75% of CGM values were within 15mg/dL, and 51% were within 10mg/dL, compared to serum glucose. MAD was 11.6mg/dL, and MARD was 10.6%. CGM values did not differ from serum glucose (P=.74). By Clarke error grid analysis, 100% of paired values were in clinically acceptable zones. By surveillance error grid analysis, 96% of paired values were within clinically acceptable agreement. CGM is a reliable tool in monitoring glycemic control in the immediate postoperative period following pancreatectomy with IAT in children.

Retrospective Analysis of Continuous Glucose Monitoring Data With the Surveillance Error Grid Supports Nonadjunctive Dosing Decisions.

Objective:The objective was to assess clinical risks of inaccurate continuous glucose monitoring (CGM) system readings as estimated by the surveillance error grid (SEG).Methods:Values from Dexcom G4 Platinum system with an advanced algorithm (Software 505) were plotted on the SEG with temporally matched reference venous (YSI) values collected during clinic visits on days 1, 4, and 7 of sensor wear. Data from a pediatric study (N = 79, age [mean ± SD] 12.2 ± 4.6 years, all with type 1 diabetes) and an adult study (N = 51, age 46.7 ± 15.8 years, 44 with type 1 diabetes and 7 with type 2 diabetes) were used.Results:Pediatric data included 2262 paired points, of which 1990 (88.0%) were in the “no risk” zone. Adult data included 2263 paired points, of which 2056 (90.9%) were in the “no risk” zone. Performance was best on Day 4, when 92.7% and 93.3% of points from the pediatric and adult studies, respectively, were in the “no risk” zone. Nine of the 4525 points (<0.2%) from 5 different sensors were in zones representing moderate risk, and none were in zones representing great or extreme risk.Conclusions:SEG analysis suggests that in pediatric and adult subjects with diabetes, using CGM values for diabetes management poses minimal risk to the user. CGM users also benefit from glucose trends and alerts.

A Post-Marketing Surveillance Study to Evaluate Performance of the EXIMO™ Blood Glucose Monitoring System.

The performance of Blood Glucose Monitoring System (BGMS) is critical as the information provided by the system guide the patient or health care professional in making treatment decisions. However, besides evaluating accuracy of the BGMS in laboratory setting, it is equally important that the intended users (healthcare professionals and patients) should be able to achieve blood glucose measurements with similar level of high accuracy. To assess the performance of EXIMO™ (Meril Diagnostics Pvt. Ltd., Vapi, Gujarat, India) BGMS as per International Organization for Standardization (ISO) 15197:2013 section 8 user performance criteria. This was a non-randomized and post-marketing study conducted at a tertiary care centre of India. A total of 1005 patients with diabetes themselves performed fingertip blood glucose measurement using EXIMO™ BGMS. Immediately after capillary blood glucose measurement using the blood glucose monitoring system, venous blood sample from each patient was obtained by a trained technician which was assessed by reference laboratory method- Cobas Integra 400 plus (Roche Instrument Centre, Rotkreuz, Switzerland). All the blood glucose measurements assessed by EXIMO™ were compared with laboratory results. Performance of the system was assessed as per ISO 15197:2013 criteria using Bland-Altman plot, Parkes-Consensus Error Grid (CEG) and Surveillance Error Grid analyses (SEG). A total of 1005 patients participated in the study. Average age of the patients was 44.93±14.65 years. Evaluation of capillary fingertip blood glucose measurements demonstrated that 95.82% measurements fulfilled ISO 15197:2013 section 8 user performance criteria. All the results lie within clinically non-critical zones; Zone A (99.47%; n=1000) and Zone B (0.53%; n=05) of the CEG analysis. As per SEG analysis, majority of the results fell within "no-risk" zone (risk score 0 to 0.5; 90.42%). The result of the study confirmed that intended users are able to obtain accurate glucose measurements when operating EXIMO™ BGMS, given only the instructions and training materials routinely provided with the system, in clinical practice.

Accuracy of Continuous Glucose Monitoring in Patients After Total Pancreatectomy with Islet Autotransplantation.

Among postsurgical and critically ill patients, malglycemia is associated with increased complications. Continuous glucose monitoring (CGM) in the inpatient population may enhance glycemic control. CGM reliability may be compromised by postsurgical complications such as edema or vascular changes. We utilized Clarke Error Grid (CEG) and Surveillance Error Grid (SEG) analysis to evaluate CGM performance after total pancreatectomy with islet autotransplantation. This subanalysis evaluated Medtronic Enlite 2 CGM values against YSI serum glucose in seven post-transplant patients (86% female; 38.6 ± 9.4 years) on artificial pancreas for 72 h at transition from intravenous to subcutaneous insulin. Sensor recalibration occurred for absolute relative difference (ARD) ≥20% x2, ≥30% x1, or by investigator discretion based on trend. Sensor analysis showed mean absolute relative difference (MARD) of 11.0% ± 11.5%. The sensors were recalibrated 8.3 times/day; active sensor was switched 1.4 times/day. Calibration factor was 7.692 ± 3.786 mg/nA·dL (target = 1.5-20 mg/nA·dL). CEG analysis showed 86.1% of pairs in Zone A (clinically accurate zone) and 99.4% of pairs in Zones A + B (low risk of error). SEG analysis of hypoglycemia/hyperglycemia risk showed 92.22% of pairs in the "no risk" zone, 5.96% of pairs in the "slight lower" risk zone, 1.01% of pairs in the "slight higher" risk zone, and only 0.81% of pairs in the "moderate lower" risk zone. Overall performance of the Medtronic Enlite 2 CGM in the post-transplant population was reasonably good with "no risk" or "slight lower" risk by SEG analysis and high CGM-YSI agreement by CEG analysis; however, frequent recalibrations were required in this intensive care population.

Performance of two updated blood glucose monitoring systems: an evaluation following ISO 15197:2013

Objective For patients with diabetes, regular self-monitoring of blood glucose (SMBG) is essential to ensure adequate glycemic control. Therefore, accurate and reliable blood glucose measurements with SMBG systems are necessary. The international standard ISO 15197 describes requirements for SMBG systems, such as limits within which 95% of glucose results have to fall to reach acceptable system accuracy. The 2013 version of this standard sets higher demands, especially regarding system accuracy, than the currently still valid edition. ISO 15197 can be applied by manufacturers to receive a CE mark for their system.Research design and methods This study was an accuracy evaluation following ISO 15197:2013 section 6.3 of two recently updated SMBG systems (Contour* and Contour TS; Bayer Consumer Care AG, Basel, Switzerland) with an improved algorithm to investigate whether the systems fulfill the requirements of the new standard. For this purpose, capillary blood samples of approximately 100 participants were measured with three test strip lots of both systems and deviations from glucose values obtained with a hexokinase-based comparison method (Cobas Integra† 400 plus; Roche Instrument Center, Rotkreuz, Switzerland) were determined. Percentages of values within the acceptance criteria of ISO 15197:2013 were calculated. This study was registered at clinicaltrials.gov (NCT02358408).Main outcome Both updated systems fulfilled the system accuracy requirements of ISO 15197:2013 as 98.5% to 100% of the results were within the stipulated limits. Furthermore, all results were within the clinically non-critical zones A and B of the consensus error grid for type 1 diabetes.Conclusions The technical improvement of the systems ensured compliance with ISO 15197 in the hands of healthcare professionals even in its more stringent 2013 version. Alternative presentation of system accuracy results in radar plots provides additional information with certain advantages. In addition, the surveillance error grid offers a modern tool to assess a system’s clinical performance.

Performance Evaluation of Three Blood Glucose Monitoring Systems Using ISO 15197: 2013 Accuracy Criteria, Consensus and Surveillance Error Grid Analyses, and Insulin Dosing Error Modeling in a Hospital Setting.

Blood glucose monitoring is an essential component of diabetes management. Inaccurate blood glucose measurements can severely impact patients' health. This study evaluated the performance of 3 blood glucose monitoring systems (BGMS), Contour® Next USB, FreeStyle InsuLinx®, and OneTouch® Verio™ IQ, under routine hospital conditions. Venous blood samples (N = 236) obtained for routine laboratory procedures were collected at a Spanish hospital, and blood glucose (BG) concentrations were measured with each BGMS and with the available reference (hexokinase) method. Accuracy of the 3 BGMS was compared according to ISO 15197:2013 accuracy limit criteria, by mean absolute relative difference (MARD), consensus error grid (CEG) and surveillance error grid (SEG) analyses, and an insulin dosing error model. All BGMS met the accuracy limit criteria defined by ISO 15197:2013. While all measurements of the 3 BGMS were within low-risk zones in both error grid analyses, the Contour Next USB showed significantly smaller MARDs between reference values compared to the other 2 BGMS. Insulin dosing errors were lowest for the Contour Next USB than compared to the other systems. All BGMS fulfilled ISO 15197:2013 accuracy limit criteria and CEG criterion. However, taking together all analyses, differences in performance of potential clinical relevance may be observed. Results showed that Contour Next USB had lowest MARD values across the tested glucose range, as compared with the 2 other BGMS. CEG and SEG analyses as well as calculation of the hypothetical bolus insulin dosing error suggest a high accuracy of the Contour Next USB.

A Multicenter Performance Evaluation of a Blood Glucose Monitoring System in 21 Leading Hospitals in Spain.

Self-monitoring of blood glucose is crucial for the effective self-management of diabetes. The present study evaluated the accuracy of the Contour® XT blood glucose monitoring system (BGMS) compared to the reference method in a large multicenter study under routine lab conditions at each hospital site. This study was conducted at 21 leading hospitals in Spain using leftover whole blood samples (n = 2100). Samples were tested with the BGMS using 1 commercial strip lot and the local laboratory hexokinase method. BGMS accuracy was assessed and results were compared to ISO 15197:2013 accuracy limit criteria and by using mean absolute relative difference analysis (MARD), consensus (Parkes) error grid (CEG), and surveillance error grid analyses (SEG). Pooled analysis of 2100 measurements from all sites showed that 99.43% of the BGMS results were within the ranges accepted by the accuracy limit criteria. The overall MARD was 3.85%. MARD was 4.47% for glucose concentrations < 70 mg/dL and 3.81% for concentrations of 70-300 mg/dL. In CEG, most results (99.8%) were within zone A ("no effect on clinical action"); the remaining ones (0.2%) were in zone B ("little to no effect on clinical action"). The SEG analysis showed that most of the results (98.4%) were in the "no risk" zone, with the remaining results in the "slight, lower" risk zone. This is the largest multicenter study of Contour XT BGMS to date, and shows that this BGMS meets the ISO 15197:2013 accuracy limit criteria under local routine conditions in 21 leading Spanish hospitals.

Mapping the Spatial Distribution of a Disease-Transmitting Insect in the Presence of Surveillance Error and Missing Data

SummaryMaps of the distribution of epidemiological data often ignore surveillance error or possible correlations between missing information and outcomes. We analyse presence–absence data at the household level (12050 points) of a disease-carrying insect in Mariano Melgar, Peru, collected as part of the Arequipan Ministry of Health's efforts to control Chagas disease. We construct a Bayesian hierarchical model to locate regions that are vulnerable to under-reporting due to surveillance error, accounting for variability in participation due to infestation status. The spatial correlation in the data allows us to identify relative inspector sensitivity and to elucidate the relationship between participation and infestation. We show that naive estimates of prevalence would be biased by surveillance error and missingness at random assumptions. We validate our results through simulations and observe how randomized inspector assignments may improve prevalence estimates. Our results suggests that bias due to imperfect observations and missingness at random can be assessed and corrected in prevalence estimates of spatially auto-correlated binary variables.

Acute Versus Chronic Injury in Error Grids

Klonoff et al recently described a new glucose meter error grid.1 This error grid updates the limits for the various zones that contain clinically significant glucose meter errors. To determine these limits, clinicians were provided scenarios and were asked to describe glucose error levels that would prompt them to treat patients in several ways ranging from no treatment to emergency treatment. One could view this exercise as clinicians responding to a patient’s symptoms, or the threat of acute injury. Yet diabetes is a disease that includes the possibilities of acute injury and chronic injury due to persistent increases in glucose. Diabetic retinopathy is an example of serious chronic injury to patients with diabetes. Hemoglobin A1c levels that exceed 5.5% are associated with diabetic retinopathy.2 Yet an A1c level of 5.5% is equivalent to a mean glucose of 111 mg/dL.3 In error grid terms, a meter measuring glucose with a true level of 111 mg/dL, but reading 100 mg/dL, demonstrates an error in the A zone, which is the no-treatment-needed zone (this is the case for all of the popular glucose error grids including Clarke,4 Parkes,5 as well as the new surveillance error grid1). To be fair, the 5.5% level of A1c is the starting point for diabetic retinopathy but for an A1c level of 6.5% (equivalent to a mean glucose of 140 mg/dL), the prevalence of diabetic retinopathy doubles to 20%, yet this level of glucose bias is still in the A zone. One might argue that the possibility of a 10% to 40% consistent bias in modern-day glucose meters is unlikely. Yet, meters can exhibit biases due to interferences. Moreover, a large source of error for any assay, including glucose meters, is lot-to-lot reagent variability. Both of these errors can contribute to produce a consistent or fixed bias. Clearly one needs to inform about allowable glucose meter deviations associated with the treatment of acute injury and the proposed error grid does that superbly. We propose that another error grid is needed to inform about diabetes complications by providing allowable error limits for long-term bias. This situation has analogies in other areas such as preventive cardiology. An assay such as low density lipoprotein (LDL) cholesterol, which is not used to diagnose acute injury, might have limits set for allowable long-term bias to inform about the risk of coronary events associated with an incorrect LDL measurement.

Error model estimation for airborne beacon‐based surveillance

A next-generation aircraft collision avoidance system under development is designed to reduce unnecessary alerts while reducing the risk of mid-air collision. Current and future collision avoidance systems rely upon range, bearing and altitude information from airborne, beacon-based surveillance. Realistic models of surveillance error characteristics are needed in order to develop and certify the safety and performance of these systems. Prior models of bearing measurements assumed uncorrelated Gaussian noise. This study explains how hidden Markov models can be used to capture the time-correlated error characteristics exhibited by recorded surveillance data. Simulations show the impact of this higher-fidelity model on predicted collision avoidance performance.

Computing the surveillance error grid analysis: procedure and examples.

The surveillance error grid (SEG) analysis is a tool for analysis and visualization of blood glucose monitoring (BGM) errors, based on the opinions of 206 diabetes clinicians who rated 4 distinct treatment scenarios. Resulting from this large-scale inquiry is a matrix of 337 561 risk ratings, 1 for each pair of (reference, BGM) readings ranging from 20 to 580 mg/dl. The computation of the SEG is therefore complex and in need of automation. The SEG software introduced in this article automates the task of assigning a degree of risk to each data point for a set of measured and reference blood glucose values so that the data can be distributed into 8 risk zones. The software's 2 main purposes are to (1) distribute a set of BG Monitor data into 8 risk zones ranging from none to extreme and (2) present the data in a color coded display to promote visualization. Besides aggregating the data into 8 zones corresponding to levels of risk, the SEG computes the number and percentage of data pairs in each zone and the number/percentage of data pairs above/below the diagonal line in each zone, which are associated with BGM errors creating risks for hypo- or hyperglycemia, respectively. To illustrate the action of the SEG software we first present computer-simulated data stratified along error levels defined by ISO 15197:2013. This allows the SEG to be linked to this established standard. Further illustration of the SEG procedure is done with a series of previously published data, which reflect the performance of BGM devices and test strips under various environmental conditions. We conclude that the SEG software is a useful addition to the SEG analysis presented in this journal, developed to assess the magnitude of clinical risk from analytically inaccurate data in a variety of high-impact situations such as intensive care and disaster settings.

The Surveillance Error Grid

Currently used error grids for assessing clinical accuracy of blood glucose monitors are based on out-of-date medical practices. Error grids have not been widely embraced by regulatory agencies for clearance of monitors, but this type of tool could be useful for surveillance of the performance of cleared products. Diabetes Technology Society together with representatives from the Food and Drug Administration, the American Diabetes Association, the Endocrine Society, and the Association for the Advancement of Medical Instrumentation, and representatives of academia, industry, and government, have developed a new error grid, called the surveillance error grid (SEG) as a tool to assess the degree of clinical risk from inaccurate blood glucose (BG) monitors. A total of 206 diabetes clinicians were surveyed about the clinical risk of errors of measured BG levels by a monitor. The impact of such errors on 4 patient scenarios was surveyed. Each monitor/reference data pair was scored and color-coded on a graph per its average risk rating. Using modeled data representative of the accuracy of contemporary meters, the relationships between clinical risk and monitor error were calculated for the Clarke error grid (CEG), Parkes error grid (PEG), and SEG. SEG action boundaries were consistent across scenarios, regardless of whether the patient was type 1 or type 2 or using insulin or not. No significant differences were noted between responses of adult/pediatric or 4 types of clinicians. Although small specific differences in risk boundaries between US and non-US clinicians were noted, the panel felt they did not justify separate grids for these 2 types of clinicians. The data points of the SEG were classified in 15 zones according to their assigned level of risk, which allowed for comparisons with the classic CEG and PEG. Modeled glucose monitor data with realistic self-monitoring of blood glucose errors derived from meter testing experiments plotted on the SEG when compared to the data plotted on the CEG and PEG produced risk estimates that were more granular and reflective of a continuously increasing risk scale. The SEG is a modern metric for clinical risk assessments of BG monitor errors that assigns a unique risk score to each monitor data point when compared to a reference value. The SEG allows the clinical accuracy of a BG monitor to be portrayed in many ways, including as the percentages of data points falling into custom-defined risk zones. For modeled data the SEG, compared with the CEG and PEG, allows greater precision for quantifying risk, especially when the risks are low. This tool will be useful to allow regulators and manufacturers to monitor and evaluate glucose monitor performance in their surveillance programs.