Assessment of Accuracy of Waterfall Plot Representations of Response Rates in Cancer Treatment Published in Medical Journals

Abstract

Response rates are a well-recognized outcome of clinical trials and provide an objective measure of drug activity. To quantify the difference between objective response rate and visual representation of response in waterfall plots in recent articles in major medical journals and to assess the change in frequency over time with which waterfall plots are used. In a cross-sectional study, original articles of 6 top journals between July 2016 and June 2018 were manually reviewed to identify articles including a waterfall plot to describe a treatment effect of cancer therapy. Response rates visually represented in waterfall plots were compared with response rates reported as study outcomes. The number of original articles with a waterfall plot as a percentage of total original articles was evaluated, sampling articles from January, February, and March for the years 2004, 2008, 2012, 2016, and 2018. Difference between response rates depicted in waterfall plots and response rates reported as study outcomes. One hundred twenty-six articles were selected for analysis. Of the 97 articles reporting investigator-assessed response rates, waterfall plots showed response rates a median (interquartile range) of 6.1% (1.8%-12.0%) higher than rates derived from investigator assessment. Forty-two articles reported response rates based on central assessment as an outcome, and waterfall plots showed response rates a median (interquartile range) of 12.0% (7.7%-18.5%) higher compared with centrally assessed response rates. The estimated percentage of original articles using waterfall plots increased from 0% in 2004 to 7% in 2018. This study suggests that waterfall plots are becoming more common in oncology literature. Waterfall plots provide a visual overestimate of response rate of cancer therapies and should be evaluated with caution.