Definition of Clinical Outcomes in Pediatric Anesthesia Research: It Is Like the Tower of Babel!

Abstract

One of the fundamental requirements of clinical research is a clear, a priori description of variables to be collected to answer the proposed research question.1 The process by which a concept or phenomenon will be measured is referred to as operationalization, a procedure which compels the researcher to make decisions about specific parameters for each variable.2 Operationalizing helps to provide a clear, objective definition of even the most complex variables thus making it easier for future investigators to replicate a study and check for reliability. Lack of operational definition of terms or inconsistent definition of the same concept may lead to nonreplicable outcomes thus making comparisons across studies extremely difficult, if not impossible.2 In a 2005 thought-provoking essay, Ioannidis3 put biomedical research firmly in the cross-hairs of public scrutiny by proclaiming that “most published research findings are false.” The central premise of the review by Ioannidis3 was that the results of a vast majority of studies are unlikely to be replicable for a variety of reasons. Some of these reasons included flexibility in designs, definitions, outcomes, and statistical analyses.3 Ioannidis3 concluded that, when there is inconsistency in definitions across studies, findings are likely to be nonreproducible. Put another way, we will always obtain different results if we are not measuring the same things. This article has since generated a clarion call for increased transparency in every aspect biomedical research.4–7 Others have suggested that biomedical investigators should use this “crisis of replication” as an opportunity for introspection by calling for a culture of transparent, open science where investigators are speaking the same language.4,5 Despite the growing call for standardization of variables and the demand for comparative clinical trials across many biomedical specialties,5–7 clinical diagnoses and reporting of many pediatric perioperative outcomes remain largely dependent on individual local expertise and observations rather than on evidence derived from carefully designed clinical trials that used standardized outcome definitions. There remains an unfaltering lack of consensus on definitions of several criteria and terminologies across many domains in pediatric anesthesia practice. In many situations, different terms or definitions are used for the same idea or concept by investigators8,9 and even by the same investigators across subsequent publications.10,11 These variabilities in definitions create a number of problems that can hamper various aspects of perioperative care. CONSEQUENCES OF NONSTANDARDIZED VARIABLES ON PEDIATRIC ANESTHESIA Lack of universally agreed definition of variables and terminologies used in pediatric anesthesia has several negative connotations. First, variability in definitions of outcome or exposure variables creates a problem for translation into clinical practice. Second, it is difficult to compare or combine data from various studies when variable end points are used across studies. Lack of consistent reporting also diminishes the possibility of conducting reliable, high-quality meta-analyses.12 When trying to determine what variables to measure at the design stage of a study, operational definition of terms is the most important step, because poor, inconsistent definitions produce meaningless outcomes that are likely to be nonreproducible. Because reproducibility is one of the fundamental assumptions in science,12 any part of the scientific process that threatens reproducibility needs to be addressed. Lack of consensus definition for many clinical variables measured in pediatric perioperative research also creates legal and public image issues for the specialty of anesthesiology. To this end, it is essential to ensure that study variables are clear, meaningful, reproducible, and relevant. To our knowledge, no prior such discussions about the consequences of nonstandard variable definition exists in the pediatric anesthesia literature. Here, we discuss the clinical and research drawbacks of this lack of consensus, illustrating with common examples. We will discuss some commonly studied outcomes: perioperative respiratory adverse events (PRAE), anesthesia-related neurobehavioral defects, and upper respiratory tract infection (URI). PERIOPERATIVE RESPIRATORY ADVERSE EVENTS PRAE are important outcomes in pediatric anesthesia.9 In the majority of cases, these are readily managed, and patients suffer no lasting sequelae. However, PRAE can lead to significant harm and is often cited as one of the commonest causes of anesthesia-related cardiac arrest in children.9 Probably the earliest use of PRAE as a composite variable was in a 2001 article by Tait et al.13 These investigators defined the components of PRAE as well as severity of each component. Though frequently used in several publications,8,9,13 PRAE is a composite variable with tenuous clinical, physiological, anatomical, or unifying pathological bases. It is a conceptual definition, and concepts are usually not directly measurable. In many publications, PRAE is derived during the analysis stage of the study. Even when prospectively defined as part of a clinical trial or cohort study, PRAE is still a derived variable, usually through the concatenation of other variables such as laryngospasm, bronchospasm, desaturation, airway obstruction, severe coughing, and stridor.14 Despite widespread usage, there is no standard definition for PRAE in the pediatric anesthesia literature. Most investigators include the variables noted above, but there is no consistency in the interpretation or usage of the term PRAE.8,9,13,14 Much has been written about the limitations of using aggregate or composite variables,15 but some of these issues are highlighted here. First, what is a composite variable and why use it? A composite outcome or end point is a variable derived from the combination of ≥2 variables such that occurrence of any 1 component of the composite indicates the occurrence of the derived variable.15,16 For example, PRAE is said to be present whenever any of its 6 components (laryngospasm, bronchospasm, oxygen desaturation, coughing, stridor, or airway obstruction) occurs in a patient.9,13,14 In this instance, using a composite end point means that more events will be observed than would for any 1 component: more patients will be classifiable as having PRAE than say bronchospasm or laryngospasm. With increasing safety of anesthesia care, the risk of having a serious adverse event has decreased over time, meaning that to study these rare but serious events (eg, intraoperative bronchospasm, or death), a clinically and economically prohibitive sample size will be required. Therefore, using a composite outcome allows for a more efficient and cost-effective study design because it enables studies to be completed with smaller sample sizes (due to higher event rates), but this comes with a price: there is loss of precision in the interpretation of the treatment effect.12,15 Problems With Composite Outcome Variables There are several key assumptions inherent in the derivation and interpretation of a good composite variable.16 First, individual components of the composite should have similar clinical significance to patients and providers, and they should occur with nearly equal frequency.12,16 Undoubtedly, mild desaturation, airway obstruction, and cough are of lower clinical significance than laryngospasm, bronchospasm, or postoperative stridor. Therefore, lumping all these variables into the composite variable—PRAE as is commonly done8,9,13 violates one of the key assumptions of a good composite variable.12,16 Second, the treatment or intervention being examined should have similar (not necessarily the same) effect on the components of the composite.15,17 Additionally, one should consider whether the biology of the individual components of the composite is similar enough for one to expect similar treatment effect or relative risk reduction.12 If these assumptions are violated, a treatment benefit ascribed to the composite variable may be (erroneously) presumed to apply to all of its components. Table. - Frequency of Perioperative Respiratory Adverse Events in a Hypothetical Steroid “Trial” Events Placebo (N = 300) Steroid (N = 302) Odds Ratio (95% CI) P Value Desaturation 68 36 2.1 (1.3–3.3) .001 Upper airway obstruction 76 40 2.2 (1.4–3.4) <.001 Laryngospasm 33 30 1.1 (0.6–1.9) .691 Bronchospasm 3 1 3.0 (0.3–29.4) .372 Cough 27 11 2.6 (1.2–5.3) .007 Stridor 7 6 2.6 (0.9–7.1) .06 PRAE 126 72 2.3 (1.6–3.3) <.001 Abbreviations: CI, confidence interval; PRAE, perioperative respiratory adverse events. For example, let us take a hypothetical randomized, placebo-controlled “trial” in which an investigator seeks to determine the efficacy of preoperative oral dexamethasone treatment (administered the night before surgery) on the risk of PRAE. The hypothetical study recruits 602 children undergoing general anesthesia (GA). Using a balanced 1:1 randomization, the investigator obtains the results detailed in the Table. It may be concluded that compared to those receiving dexamethasone, children receiving placebo were 2.3 times more likely to develop PRAE. However, steroid pretreatment was not uniformly efficacious across all the components of PRAE in this example. Indeed, only 4 subjects in the this “trial” developed bronchospasm (for which there is biological plausibility of steroid efficacy). The main contributors to the increased odds ratio in the trial were coughing, upper airway obstruction, and desaturation indicating that the treatment did not have comparable effects on the events in the composite. Although the composite as a whole appear to have been affected by the treatment, there was no evidence of benefit relating to its most important constituents (laryngospasm and bronchospasm). Therefore, PRAE would not have being a good composite outcome in this “trial,” and the “findings and conclusions” must be interpreted with caution. DEFINING PEDIATRIC PRAE Given the prevailing inconsistencies in the definition of pediatric perioperative respiratory complications and the issues associated with such a deficiency, there is an urgent need to come up with a workable and agreeable definition of PRAE. Few will disagree with the assertion that reaching a consensus on what constitutes clinically important measures of pediatric perioperative respiratory complications is valuable to patients, clinicians, regulators and payers. Appreciating what stakeholders consider important is important to contextualize measured outcome variables. All currently reported components of PRAE do not carry the same weight in terms of risk of progression to serious morbidity and/or mortality.8,9,13,18 Whereas many clinicians may view oxygen desaturation of 94% as a mere, easily correctable event, none will ignore laryngospasm or bronchospasm because of the potential for these clinical diagnoses to spiral out of control. Similarly, “upper airway obstruction” is an imprecise term that can indicate perioperative retro-lapsed tongue, croup, laryngospasm, or upper airway swelling. By aggregating these multiple primary end points, investigators have implicitly given equivalent weight to the health burden of each of the adverse respiratory events, when the burden is unequivocally different. Due to the foregoing, we suggest that desaturation and upper airway obstruction should be excluded from the composite definition of PRAE. Given the importance of pediatric perioperative respiratory complications and the limitations of the composite variables currently used to define them, the logical question is, what is the alternative and how can we standardize these outcome variables? Several methods to define clinically important outcome variables have been utilized by other medical disciplines.19 These include use of focus groups, use of clinical vignettes, use of semistructured interviews embedded in clinical trials or conducted independent of clinical trials, use of weighted approach based on perceived severity and seriousness, and use of Delphi panels.20 Each of these approaches have their own limitations. For example, the popular Delphi method is time-consuming and expensive, and there is no standardized method for carrying out a Delphi panel consensus.20 Another possible solution may be to have journal editorial boards determine the acceptable definitions of selected variables for publication. Recent efforts by the British Journal of Anaesthesia (BJA) is a step in this direction. In 2015, the BJA sponsored a meeting of experienced perioperative investigators to create a consensus process of standardizing end points in perioperative medicine.21 This “Standardized Endpoints for Perioperative Medicine” (StEP) Group was charged with providing expert, evidence and consensus-based guidelines for selected outcomes in adult perioperative medicine. However, this approach has the potential of being criticized as editorial imperialism. Regardless of which of the preceding options is chosen to reach a consensus definition for PRAE, we stress that international collaborative efforts are needed. Also, if a composite definition is going to be proposed by a panel of experts on PRAE, the requirements for a good composite variable must be borne in mind.15,16 We suggest that a definition that includes an intervention, such as use of succinylcholine, propofol, or positive pressure ventilation for laryngospasm, has clinical relevance and is intuitively valid. Similar case can be made for the use of intraoperative or recovery room albuterol for bronchospasm, because use of albuterol in these settings (as opposed to preoperative premedication) is an indication of therapeutic intervention rather than prophylaxis. We suggest that including an active treatment to prevent a serious outcome into the definition of laryngospasm or bronchospasm is both clinically relevant and should appeal to all stakeholders (patients/families, caregivers, and payers). OTHER COMMON NONSTANDARDIZED PERIOPERATIVE OUTCOME VARIABLES Although the present viewpoint has so far focused primarily on PRAE given its frequency and clinical importance, there are several other examples of unstandardized variables in pediatric anesthesia. One area of research that continues to generate controversies and remains highly relevant is the question of anesthesia-related neurotoxicity.22 Whether children who undergo GA in their early years are at risk of long-term neurocognitive abnormalities continues to be relevant because millions of children undergo procedures requiring GA every year in the United States and around the world. Unfortunately, despite the clinical and socio-epidemiologic importance of the subject of neurotoxicity, to date there is no consensus on the definition of the outcome variable.22 Diagnostic criteria for anesthesia-related long-term neurobehavioral deficits have varied widely, and many studies have used a variety of diagnostic instruments making comparison between them extremely difficult. Unsurprisingly, evidence of causality has been mixed, largely due to lack of consistency in exposure and outcome variable definition.22 For example, which component of the battery of neurodevelopmental tests should we be measuring? Is verbal aptitude more important than global performance intelligence quotient? Or are tests of memory, language, attention, motor, and executive function more important? These are complex questions and have remained largely unanswered. Another commonly encountered clinical conundrum is pediatric URI and whether to proceed with anesthesia and elective surgery. Indeed, URI is the commonest, nonpreventable cause of surgical cancellation and it is often associated with perioperative respiratory complications.23 Despite these, definition of URI is nonstandardized. For most studies, URI is said to be present when ≥2 of the following respiratory symptoms/signs are present: rhinorrhea, sore or scratchy throat, sneezing, nasal congestion, malaise, cough, or fever ≥38°F together with confirmation by a parent.24 Thus, URI is a composite variable derived from a combination of symptoms/signs with varying frequencies.24 The limitations of composite variable detailed above also applies here. Given that URI is often used as an exposure variable for PRAE (the outcome variable), it can be surmised that the potential for misclassification bias is multiplicative. CONCLUDING THOUGHTS Recognizing the truth in the aphorism that “to measure something, you first have to define it,” we make an urgent call for a standardized definition of pediatric perioperative outcomes in general and argue that a good place to begin is to formulate a workable, sound, and robust definition of PRAE. International acceptance of chosen definition for PRAE is critical, to facilitate research collaboration, support for national and international perioperative trials, and effective communication at all levels. We also concede the fact that virtually everything in biomedical research suffers from inadequate definitions. For example, even “objective data” such as hospital length of stay, intensive care unit (ICU) length of stay, ventilator days, and hypotension all suffer from variability due to differences people may have with discharge criteria, criteria for extubation, and what constitutes hypotension. However, until common vocabularies for pediatric perioperative outcomes are agreed on, communication and research in pediatric anesthesia will continue to be reminiscent of events after the proverbial Tower of Babel. The Tower of Babel in the Old Testament described how God confounded humanity by making them move from one common language to a multiplicity of languages so that communication became nearly impossible. Just like in ancient biblical days, the existence of multiple variable definitions creates communication challenges in contemporary biomedical literature. For a technical term to have value, it must have consistency in interpretation and usage. PRAE in its current usage fails in this regard and should be abandoned because it risks erroneous assumptions.8,9 Similarly, terms like “neurotoxicity” and “URI” should probably be avoided except when they are being used as loose generalization, not as specific research-based outcome measures. We acknowledge that the task of coming up with a widely acceptable standardized perioperative outcome definition is so daunting that inertia may be a more appealing option. However, the apparent enormity of a problem is insufficient excuse for not undertaking constructive efforts toward a workable solution. Here we make a modest attempt toward a solution by taking the first necessary step: recognizing that the problem exists and proffering modest solutions. DISCLOSURES Name: Olubukola O. Nafiu, MD, FRCA, MS. Contribution: This author helped with study idea, concept, and manuscript preparation. Name: Joseph D. Tobias, MD. Contribution: This author helped with idea, manuscript preparation, and editing. Name: James A. DiNardo, MD. Contribution: This author helped with idea, manuscript preparation, and editing. This manuscript was handled by: Thomas R. Vetter, MD, MPH.