Limits to the analysis and interpretation of non-experimental clinical data

Abstract

The above quotation, while made in a slightly different context, is worthy of further examination as it applies to papers by Van Putten et al. [2] and Skladowski et al. [3] in the current issue of this journal. Both papers present analyses of treatment time-dependent variables on local tumor control of larynx cancer based on detailed retrospective analyses of large clinical data sets. While these papers are the primary stimulus for the editor's decision to invite editorial comment, they share many characteristics with similar analyses of clinical data on the same subject published recently in the radiation oncology literature. Editorially, these papers provide a focus for the examination of Learner's viewpoint in relation to the retrospective analysis and interpretation of clinical data in determinations of time and dose effects on tumor control. In terms of size, the series of cases from Rotterdam and Gliwice are among the largest single institutional collections of data on larynx cancer treated by radiotherapy. Follow-up for the endpoints of local control and survival has been excellent. The raw data on disease extent and treatment parameters is complete and at least comparable with any similar retrospective analysis of larynx cancer outcome. There can be no doubt, however, that these data sets are non-experimental as the radiation treatment parameters of total dose, fraction number and overall time were never designed to explore the effects for which they were ultimately studied in retrospect. Are these data then too weak, as Leamer suggests, to make sensible inferences, or can supplementary information be provided to strengthen the inferences which are drawn? Only by reviewing such information and subjecting it to appropriate and valid statistical analysis can the relative strength or weakness of the inferences drawn from such non-experimental data sets be established. Van Putten et al. [2] state that 'a difference in local control rate was found between patients with continuous course (cc) and split course (sc), but no evidence of an effect of overall treatment time within these treatment groups'. The authors clearly acknowledge the possibility of selection bias as the explanation for the lack of an effect of overall treatment time in their data. The strength of their conclusion however, rests largely with their ability to demonstrate that bias, and in particular selection bias, has been adequately considered. The authors indicate that patients treated by surgery either up front or after a poor response after a dose of less than 50 Gy were excluded, as were a total of 242 cases treated by orthovoltage radiation prior to 1975. No details are provided regarding the total number of exclusions or their disease extent. Without such information it is impossible to assess the potential impact of such exclusions on their conclusion. This is a particular concern in view of their finding of an almost halving of the relative local failure rate during the second decade of the study. The greatest concern in relation to the analysis of clinical data of this type relates to physician bias in treatment allocation and dose prescription. While one cannot disagree with the view expressed by Van Putten et al. that 'heterogeneity in treatments is a prerequisite for any analysis of split, dose or treatment time', it must also be emphasized that the cause for such heterogeneity must be dearly understood in order to value the interpretations made from such data. The heterogeneity of data over a 20-year period in Rotterdam is considerable and clearly caused, among other things, by changes in staff, treatment philosophy, equipment, and response to treatment. Many of these factors have complex interactions and are impossible to quantify. As such, the reader cannot be assured that the conclusions drawn from this very heterogenous data set are not attributable to the causes, rather than the result of heterogeneity. An additional potential source of bias in non-experimental data may arise in the methods chosen to perform-the data analysis. A fundamental issue is the categorization