Common errors in randomized controlled trials submitted for publication to the American Journal of Orthodontics and Dentofacial Orthopedics

Abstract

The consolidated standards of reporting trials (CONSORT) guidelines have been drafted to facilitate accurate and transparent trial reporting and implicitly guide researchers to follow a good methodology.1Moher D. Hopewell S. Schulz K.F. Montori V. Gøtzsche P.C. Devereaux P.J. et al.CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials.BMJ. 2010; 340: c869Crossref PubMed Scopus (3130) Google Scholar The American Journal of Orthodontics and Dentofacial Orthopedics (AJO-DO) endorsed the CONSORT reporting guidelines in 2004 with the requirement to submit the completed CONSORT checklist.2Turpin D.L. CONSORT and QUOROM guidelines for reporting randomized clinical trials and systematic reviews.Am J Orthod Dentofacial Orthop. 2005; 128 (discussion 686): 681-685Abstract Full Text Full Text PDF PubMed Scopus (48) Google Scholar Consequently, active implementation of the guidelines has been adopted, and restructuring of the article format with the inclusion of multiple subheadings to facilitate accurate and transparent reporting at the submission stage.3Pandis N. Shamseer L. Kokich V.G. Fleming P.S. Moher D. Active implementation strategy of CONSORT adherence by a dental specialty journal improved randomized clinical trial reporting.J Clin Epidemiol. 2014; 67: 1044-1048Abstract Full Text Full Text PDF PubMed Scopus (61) Google Scholar,4Koletsi D. Fleming P.S. Behrents R.G. Lynch C.D. Pandis N. The use of tailored subheadings was successful in enhancing compliance with CONSORT in a dental journal.J Dent. 2017; 67: 66-71Crossref PubMed Scopus (16) Google Scholar Those attempts were supplemented with 2 explanatory articles for the authors, which can be found at the AJO-DO Web site. Despite improvements in reporting, some submissions still fail to fully adhere to the guidelines. In this article, we aim to highlight common deficiencies in terms of reporting and methodology in the relevant sections to help prospective authors when preparing their submissions. The Table summarises the key points.TableCommon errors in submitted randomized controlled trialsItemCommon errorsGeneral format, title, and introduction GeneralAJO-DO randomized controlled trial format with subheadings not followed TitlePICO elements and randomized trials not included BackgroundNoninclusion of a systematic review or prior evidence to justify the need of the new trialMethods Trial designEspecially split-mouth designs are often not provided Participants, eligibility criteria, and settingsSome details are often missing Interventions OutcomesConfusion between outcome and outcome measures Sample size calculationNot all necessary items for replication are reportedAssumptions are often over-optimistic, resulting in small sample sizes RandomizationOften not well described, and confusion about simple and restricted randomization methods BlindingMethods to reduce bias because of blinding is often not described Statistical analysisClustering ignoredToo many comparisons of limited clinical relevanceTesting within treatment groupsSuboptimal analysis of longitudinal data such as testing per time point in the presence of longitudinal dataResults Flow diagramIncorrectly filled out flow diagram and confusion as to whether ITT or PP analysis was implemented Baseline dataA limited number of baseline characteristicsStatistical comparisons for baseline characteristics between arms Number analyzed for each outcome, estimation, and precision, subgroup analysesNot clear how many patients were analyzedEstimates and confidence intervals are not reported with emphasis on P valuesP values presented as <0.05 or >0.05Multiple comparisonsSelective reportingExact P values not reportedStandard deviations presented as ±Standard errors were reported instead of standard deviationsThe correlation coefficient and the intracluster correlation coefficient are missing from split-mouth and clustered designs, respectivelyDiscussion Main findingsOverinterpretation is based on statistical significance and not on clinical relevance LimitationsOften not well discussed GeneralizabilityConfusion as to the meaning of this term Open table in a new tab Some authors still use the Introduction, Methods, Results, and Discussion format instead of the AJO-DO format with subheadings. Although this is not a reason for rejection, it prolongs the process unnecessarily because the manuscript is sent back to the authors for reformatting. It is important that each subheading includes the required information accurately and transparently to the level of detail than a knowledgeable person can replicate the results whenever applicable. To ensure that a study is appropriately indexed and the research question is evident from the title, authors should use the word randomized in the title to indicate that the participants were randomly assigned to their comparison groups. The structure of the title should also follow the PICO+ (patient problem or population, intervention, comparison, and outcomes) format. The “+” indicates other relevant information regarding the design, such as single/double-blind, single/multicenter, split-mouth design. It is unethical to expose humans unnecessarily to the risks of research (Declaration of Helsinki). The need for a new trial should be justified in the Introduction and should include a reference to a systematic review of previous similar trials or a note of the absence of such trials. Justification is based on the concept of equipoise5Freedman B. Equipoise and the ethics of clinical research.N Engl J Med. 1987; 317: 141-145Crossref PubMed Scopus (1609) Google Scholar: genuine uncertainty in the scientific community as to whether an intervention is effective or not, which can be identified from the available evidence pooled in the form of a systematic review. Specific trial details such as the design (split-mouth, cluster-randomized, noninferiority, factorial, and cross-over) and allocation ratio should be stated to set the stage of what will follow. If the trial design is split-mouth,6Pandis N. Chung B. Scherer R.W. Elbourne D. Altman D.G. CONSORT 2010 statement: extension checklist for reporting within person randomised trials.BMJ. 2017; 357: j2835Crossref PubMed Scopus (106) Google Scholar cluster-randomized,7Campbell M.K. Piaggio G. Elbourne D.R. Altman D.G. CONSORT GroupConsort 2010 statement: extension to cluster randomised trials.BMJ. 2012; 345: e5661Crossref PubMed Scopus (1040) Google Scholar noninferiority,8Piaggio G. Elbourne D.R. Pocock S.J. Evans S.J. Altman D.G. CONSORT GroupReporting of noninferiority and equivalence randomized trials: extension of the CONSORT 2010 statement.JAMA. 2012; 308: 2594-2604Crossref PubMed Scopus (712) Google Scholar and cross-over,9Dwan K. Li T. Altman D.G. Elbourne D. CONSORT 2010 statement: extension to randomised crossover trials.BMJ. 2019; 366: l4378Crossref PubMed Scopus (115) Google Scholar the CONSORT extensions for those designs should be consulted. The trial eligibility criteria should be stated. Information on the settings and locations is crucial to allow clinicians to judge both the applicability and generalizability of a trial in relation to their clinical setting. The following should be stated: number (single-center or multi-center) and type of settings, care providers involved, and locations in which the study was carried out (country, city, and type of centers [eg, community, office practice, hospital clinic, or inpatient unit]). A clear description of the assigned interventions, including the control group, should be provided. The description should allow a clinician wanting to use the intervention to know exactly how to administer the intervention that was evaluated in the trial. All outcome measures, whether primary or secondary, should be identified and completely defined. Importantly, the primary outcome measure is the prespecified outcome considered to be of greatest importance to relevant stakeholders and is usually the one used in the sample size calculation. The reporting of several primary outcomes is not recommended as this results in problems of interpretation associated with a multiplicity of analyses. When outcomes are assessed at several time points after randomization, authors should also indicate the prespecified time point(s) of primary interest and the instruments (eg, validated questionnaires or scales used to collect the outcomes). Authors often confuse the actual outcome with the effect measures. For example, a bond failure is not the same as the risk of failure. The former is the outcome, while the latter refers to the percentage of failures and is a measure of effect. The authors should indicate how the sample size was determined. If a formal power calculation was used, the authors should identify the primary outcome on which the calculation was based, all the quantities used in the calculation, and the resulting target sample size per study group. It is preferable to quote the expected result in the control group and the expected difference between the groups, which can be expressed as a percentage with the event or mean for each group used in their calculations. For continuous outcomes and the mean values, the corresponding standard deviations should be provided to allow for sample size replication. The selected power and α levels should also be provided. Authors often fail to provide all required numbers to allow for replication. Finally, the assumptions must be clinically relevant based on preexisting evidence or piloting. Selecting unrealistically large differences to detect technically deflates the required sample size but may result in an inconclusive and waste trial. Recruitment of trial participants can occur over a long period. If an intervention is working particularly well or badly, the study may need to be ended early for ethical reasons. This concern can be addressed by examining results as the data accumulate, preferably by an independent data monitoring committee. This is not common in orthodontic trials. Authors should clearly specify the method of sequence generation, such as a random number table or a computerized random number generator, and the methods employed to conceal this random allocation. There is confusion between simple randomization and forms of restricted randomization. The former resembles the tossing of a coin, and although an accepted method, it is unlikely to provide an equal number of patients between treatment arms, especially in small trials common in orthodontics. Restricted randomization such as permuted blocks or minimization uses a mechanism to balance the number of patients between arms, and it is not equivalent to simple randomization. Therefore, if the authors state that simple randomization was used in a small trial, for example, with 60 patients, it is highly unlikely that the allocation ratio would be 1:1 unless investigators implemented some form or restrictions to balance the numbers between treatment groups. Allocation concealment pertains to the mechanism used to make the treatment allocation unpredictable and best implemented using an external service unrelated to the research team conducting the trial. Allocation concealment is not the same with blinding as it occurs before the treatment allocation, whereas blinding, if feasible, occurs after the delivery of the intervention. Therefore, allocation concealment is always feasible, whereas blinding is not. Reporting the randomization steps (random number generation, allocation concealment, and implementation of the randomization) is often suboptimal in the submitted manuscripts. Differential follow-up, outcome recording, and losses to follow-up associated with lack of blinding may bias trial results. To reduce this risk of bias, authors should report who was blinded in the trial (eg, patients, investigators, data assessors, and data analysts). Statistical methods should be described with enough detail to enable a knowledgeable clinician with access to the original data to verify the reported results (www.icmje.org). Standard methods of analysis assume that the data are independent. For controlled trials, this usually means that there is one observation per participant. Data analysis should be based on counting each participant once or should be done by using more complex statistical procedures. Treating multiple observations (ie, bond failures, periodontal indexes, etc) from 1 participant as independent data is a serious error, and unfortunately, a common error.10Koletsi D. Pandis N. Polychronopoulou A. Eliades T. Does published orthodontic research account for clustering effects during statistical data analysis?.Eur J Orthod. 2012; 34: 287-292Crossref PubMed Scopus (21) Google Scholar,11Fleming P.S. Koletsi D. Polychronopoulou A. Eliades T. Pandis N. Are clustering effects accounted for in statistical analysis in leading dental specialty journals?.J Dent. 2013; 41: 265-270Crossref PubMed Scopus (36) Google Scholar In the presence of longitudinal data (multiple observations over time) it is common in the submitted articles to compare groups at each time point. This approach increases the chance of false positives, encourages selective reporting, and may result in loss of power because the longitudinal structure of the data is ignored.12Mheissen S. Khan H. Almuzian M. Alzoubi E.E. Pandis N. Do longitudinal orthodontic trials use appropriate statistical analyses? A meta-epidemiological study.Eur J Orthod. 2021; : cjab069Crossref PubMed Google Scholar Ignoring such data structure can miss the evolution of the treatment effects over time. Spaghetti plots for continuous outcomes or probability plots for categorical outcomes are a good way to visualize the data, followed by appropriate but usually advanced statistical methods requiring statistical expertise. Because of the high risk for spurious findings, subgroup analyses are often discouraged, and the focus should be on the overall effect. Post-hoc subgroup comparisons (analyses done after looking at the data) are especially likely not to be confirmed by further studies, and such analyses do not have great credibility. In some studies, imbalances in participant characteristics are adjusted for using multiple regression analysis. Although the need for adjustment is much less in randomized controlled trials than in epidemiologic studies, an adjusted analysis may be sensible, especially if one or more variables are thought to be prognostic. Ideally, adjusted analyses should be prespecified in the study protocol and not decided after looking at the data. Statistical analysis reporting is often suboptimal and not well justified. Important issues arise when multiple tests are conducted, such as cephalometric measurements, which often include nonclinically relevant measurements and invoke the multiplicity problem with false-positive results that can be interpreted as important. Another problem is the inferences based on testing against the baseline. For example, in each trial arm, a pretreatment and posttreatment tests are performed, and if one is significant and the other is not, we infer that there is a difference in the effect. This approach is flawed because it does not make a direct comparison between treatment arms, which is the aim of the study, and because it has a higher chance of false-positive results, it can be misleading.13Bland J.M. Altman D.G. Comparisons against baseline within randomised groups are often used and can be highly misleading.Trials. 2011; 12: 264Crossref PubMed Scopus (113) Google Scholar A CONSORT participant flow diagram should be included, which enables the clinician to ascertain when a study took place, the period of participant recruitment, location of recruitment, the rate at which participants were recruited, and the duration of follow-up periods (minimum, maximum, and median). Any protocol deviations and/or losses at the appropriate stage should also be documented. The authors should also disclose factors extrinsic to the trial that affected the decision to stop the trial and who decided to stop the trial, including reporting the role the funding agency played in the deliberations and in the decision to stop the trial. There is confusion in terms of what is the available case, per-protocol (PP), and intention-to-treat (ITT) analysis and their variants together with the reporting of missing data. If patients are lost to follow-up, ITT analysis is not possible unless the missing outcome is imputed. Most often, the conducted analyses were either PP (patients adhering to the interventions according to the protocol) or available case (patients with any available data included), a finding confirmed in the literature.14Pandis N. Fleming P.S. Katsaros C. Ioannidis J.P.A. Dental research waste in design, analysis, and reporting: a scoping review.J Dent Res. 2021; 100: 245-252Crossref PubMed Scopus (7) Google Scholar The flow diagram often incorrectly refers to PP analysis as ITT analysis. Randomized trials aim to compare participants that differ only with respect to the intervention (treatment) and randomization prevents selection bias; any differences in baseline characteristics are expected to be the result of chance rather than bias. The study groups should be compared at baseline for important demographic and clinical characteristics so that clinicians can assess how similar they were. This data should be presented in a table, and depending on the type and distribution of the data, a measure of the central tendency and variability. Standard errors and confidence intervals are not appropriate for describing variability. Significance testing of baseline differences is neither required nor recommended as such hypothesis testing can mislead clinicians. Rather, comparisons at baseline should be based on consideration of the prognostic strength of the variables measured and the size of any chance imbalances that have occurred. Baseline tables in submitted randomized controlled trials often provide very limited information on baseline characteristics such as only age, gender distribution between arms. It is also common for statistical comparisons to be presented, and the presentation of standard errors instead of standard deviation is sometimes encountered. The number of participants in each group is an essential element of the analysis. Although the flow diagram may indicate the numbers of participants analyzed, these numbers often vary for different outcome measures. The number of participants per group should be given for all analyses. Results should not be presented solely as summary measures, such as relative risks. Expressing results as fractions also aids the clinician in assessing whether some of the randomly assigned participants were excluded from the analysis. Participants may sometimes not receive the full intervention, or some ineligible patients may have been randomly allocated in error. One widely recommended way to handle such issues is to analyze all participants according to their original group assignment, regardless of what subsequently occurred. This ITT strategy is not always straightforward to implement. It is common for some patients not to complete a study-they may drop out or be withdrawn from active treatment and thus are not assessed at the end. Conversely, analysis can be restricted to only participants who fulfill the protocol in terms of eligibility, interventions, and outcome assessment. This analysis is known as an on-treatment or PP analysis. However, excluding participants from the analysis can lead to erroneous conclusions. The ITT analysis is generally favored because, in general, it avoids bias associated with the nonrandom loss of participants. Regardless of whether authors use the term ITT, they should clarify which and how many participants are included in each analysis. Noncompliance with assigned therapy may mean that the ITT analysis underestimates the potential benefit of the treatment, and additional analyses, such as a PP analysis, may therefore be considered. However, it should be noted that such analyses could be flawed. For each outcome, study results should be reported as a summary of the outcome in each group (eg, the number of participants with or without the event and the denominators, or the mean and standard deviation of measurements), together with the contrast between the groups, known as the effect size. For binary outcomes, the effect size could be the risk ratio (relative risk), odds ratio, or risk difference; for survival time data, it could be the hazard ratio or difference in median survival time; and for continuous data, it is usually the difference in means. The mean (standard deviation) should be provided, for example, as 10 (4) and not as 10 (±4). The latter presentation is common but not recommended. In addition, the standard errors should not be presented with the means as they are inferential (not descriptive) and can be misleading as they might give a false sense of the tight spread of the data.15Lang T.A. Altman D.G. Basic statistical reporting for articles published in biomedical journals: the “Statistical Analyses and Methods in the Published Literature” or the SAMPL Guidelines.Int J Nurs Stud. 2015; 52: 5-9Crossref PubMed Scopus (147) Google Scholar To provide clinical relevance, confidence intervals should be presented for the contrast between groups. A common error is the absence of confidence intervals or the presentation of separate confidence intervals for the outcome in each group rather than for the treatment effect. P values may be provided in addition to confidence intervals, but results should not be reported solely as P values. Results should be reported for all planned primary and secondary endpoints, not just for analyses that were statistically significant or interesting. Selective reporting within a study is widespread and serious.16Dwan K. Altman D.G. Clarke M. Gamble C. Higgins J.P. Sterne J.A. et al.Evidence for the selective reporting of analyses and discrepancies in clinical trials: a systematic review of cohort studies of clinical trials.PLoS Med. 2014; 11: e1001666Crossref PubMed Scopus (111) Google Scholar The P value should not be presented as <0.05 or >0.05 but as exact numbers. Two decimal points are sufficient, and if the P = 0.000, it can be presented as <0.001. Missing items for alternate designs such as split-mouth6Pandis N. Chung B. Scherer R.W. Elbourne D. Altman D.G. CONSORT 2010 statement: extension checklist for reporting within person randomised trials.BMJ. 2017; 357: j2835Crossref PubMed Scopus (106) Google Scholar and clustered trials7Campbell M.K. Piaggio G. Elbourne D.R. Altman D.G. CONSORT GroupConsort 2010 statement: extension to cluster randomised trials.BMJ. 2012; 345: e5661Crossref PubMed Scopus (1040) Google Scholar include the correlation between arms (split-mouth) and the intracluster correlation coefficient (clustered), which are important for sample size calculation of future trials. For split-mouth designs, justification in terms of the absence of carry-across effects is often not provided. Clinicians require information about the harms and the benefits of interventions to make rational and balanced decisions. The existence and nature of adverse effects can have a major impact on whether a particular intervention will be deemed acceptable and useful, and harms are often not reported. Authors should report the overall results in the context of the existing evidence using a systematic approach and not just by including evidence supporting the study results. The clinical significance and precision of estimates should be highlighted rather than statistical significance (eg, P value). In addition, overinterpretation of results from subgroup analyses should be avoided as these are likely to have low power but may also give false-positive results. It is common for interpretation to be based on statistical testing ignoring clinical relevance. Authors should document potential sources of biases, imprecision of estimates and multiplicity, and conclusions drawn from any subgroup analyses. There is often confusion about the meaning of generalizability, and this is also referred to as external validity. This is often a matter of author judgment influenced by characteristics of the participants included in the trial, the trial setting, the treatment regimens tested, and the outcomes assessed. Conclusions drawn regarding the generalizability of the trial are underpinned by the detail provided in the Methods section (eligibility criteria, setting and location, the interventions and how they were administered, the definition of outcomes, the period of recruitment, and follow-up).