Bench-to-bedside review: The importance of the precision of the reference technique in method comparison studies – with specific reference to the measurement of cardiac output

Maurizio Cecconi,Giorgio Della Rocca,Andrew Rhodes,R Michael Grounds,Jan Poloniecki

doi:10.1186/cc7129

Maurizio Cecconi, Giorgio Della Rocca + Show 3 more

Open Access

PDF Available

https://doi.org/10.1186/cc7129

Copy DOI

Export

Save

Cite

Abstract
Highlights/Summary
Full-Text PDF
Similar Papers

Abstract

Listen

Bland-Altman analysis is used for assessing agreement between two measurements of the same clinical variable. In the field of cardiac output monitoring, its results, in terms of bias and limits of agreement, are often difficult to interpret, leading clinicians to use a cutoff of 30% in the percentage error in order to decide whether a new technique may be considered a good alternative. This percentage error of ± 30% arises from the assumption that the commonly used reference technique, intermittent thermodilution, has a precision of ± 20% or less. The combination of two precisions of ± 20% equates to a total error of ± 28.3%, which is commonly rounded up to ± 30%. Thus, finding a percentage error of less than ± 30% should equate to the new tested technique having an error similar to the reference, which therefore should be acceptable. In a worked example in this paper, we discuss the limitations of this approach, in particular in regard to the situation in which the reference technique may be either more or less precise than would normally be expected. This can lead to inappropriate conclusions being drawn from data acquired in validation studies of new monitoring technologies. We conclude that it is not acceptable to present comparison studies quoting percentage error as an acceptability criteria without reporting the precision of the reference technique.

Highlights

In 1986, Bland and Altman [1] first suggested their statistical method for assessing agreement between two measurements of the same clinical variable
Sander and colleagues [11] demonstrated that, in comparison with intermittent thermodilution (ITD), the Vigileo/Flotrac device (Edwards Lifesciences LLC, Irvine, CA, USA) had a bias of 0.6 litres per minute and limits of agreement of between –2.2 to +3.4 litres per minute. These results were reported as demonstrating that the new tested device, the Vigileo, was not a good measure of cardiac output compared with the reference technique
In order for a monitor to be used to detect this 10% change, it must have a level of Precision of the reference technique for n averaged measurements and the corresponding percentage error (PE) from the Bland-Altman plot for a fixed level of precision of the studied technique (29%)

Summary

Introduction

In 1986, Bland and Altman [1] first suggested their statistical method for assessing agreement between two measurements of the same clinical variable. These results were reported as demonstrating that the new tested device, the Vigileo, was not a good measure of cardiac output compared with the reference technique. With respect to the Bland-Altman plot, accurate measurements mean small bias and precise measurements mean narrow limits of agreement.

Results

Conclusion