Rethinking the “Evidence Pyramid”: A shift From Positivism to Pragmatism

Assessing methodological quality and biological plausibility in occupational health psychology.

Much of the methodological discussion around experiments in economics and other social sciences is framed in terms of the notions of internal and external validity. The standard view is that internal validity and external validity stand in a relationship best described as a trade-off. However, it is also commonly held that internal validity is a prerequisite to external validity. This article addresses the problem of the compatibility of these two ideas and analyzes critically the standard arguments about the conditions under which a trade-off between internal and external validity arises. Our argument stands against common associations of internal validity and external validity with the distinction between field and laboratory experiments and assesses critically the arguments that link the artificiality of experimental settings done in the laboratory with the purported trade-off between internal and external validity. We conclude that the idea of a trade-off or tension between internal and external validity seems, upon analysis, far less cogent than its intuitive attractiveness may lead us to think at first sight.

Discussion: ‘Complications of labor induction among multiparous women’ by Battista et al

VioLit summary: OBJECTIVE: The purpose of this study by Friedrich-Cofer and Huston was to provide a literature review and critique of the debate over the relationship between viewing violence on television and actual aggression and violence. This study supported the majority of the literature which claims that a relationship does exist and discussed the theoretical and social policy implications of this relationship. METHODOLOGY: This literature review was divided into two main areas of research: experimental studies (both laboratory and field studies) and correlational studies (primarily longitudinal studies). Within each of these areas, methodological issues were addressed, as well as results, trends and potential sources of bias. FINDINGS/DISCUSSION: Laboratory studies have had high internal validity because of their random sampling techniques and their ability to manipulate the independent variables. However, three major questions were raised about their external validity: the degree to which the stimuli used reflected actual television programming; the degree to which their artificial settings lead to experimenters' influencing the subjects' responses; and the degree to which the measures of aggression in the laboratory actually mirror real aggression. This study refuted these questions by explaining that a content analysis of laboratory studies showed that most use actual television programming; that aggression seemed, in most studies, to increase when experimenters' left the room not when they remained; and that a variety of measures were used in studies for aggression, many of which reflected real-life situations. Field experiments were considered a good methodological choice because they utilize the random assignment and experimental manipulation of variables in a natural setting, increasing both internal and external validity. However, questions again were raised in the same three areas of stimuli, setting and measures of aggression: there has been the problem of finding control (nonviolent) stimuli which the subjects would voluntarily watch on a regular basis; most of the settings of field research have been within residential institutions for adolescents where often their television viewing is monitored or controlled, not voluntary; when measuring aggression, problems arose in distinguishing between serious and playful aggression, as well as distinguishing between aggression related to group dynamics (within a group home setting) and individual aggression. Correlational studies have been primarily very large and involving a diverse range of subjects in terms of race, ethnicity and socioeconomic status. They have had high external validity but low internal validity. Longitudinal studies have allowed for analysis across time and have seemed to support a small but consistent positive relationship between violent behavior and violent television viewing. Problems with these studies, however, were pointed out such as the possibility of selection bias; bias caused by other variables not being statistically controlled; or self-reporting biases. This study pointed out, however, that most correlational studies have rigorously controlled for as many other variables as they could and have attempted to find as many alternative measures of aggression to self-reporting as possible. This study claimed that the literature overall supported a bi-directional model of relations between viewing television violence and aggressive behavior. In other words, there appeared to be both a tendency for violent television viewing to positively influence aggressive behavior and a tendency for violent behavior to positively influence the desire to watch violent television shows. AUTHORS' RECOMMENDATIONS: The authors suggested that social scientists must acknowledge that the dialogue about violence on television must broaden to include a discussion on societal values. They also stressed the urgent need for connecting policy issues with theoretical and research issues. (CSPV Abstract - Copyright © 1992-2007 by the Center for the Study and Prevention of Violence, Institute of Behavioral Science, Regents of the University of Colorado) KW - Television Viewing KW - Television Violence KW - Literature Review KW - Media Violence Effects KW - Aggression Causes KW - Violence Causes KW - Child Aggression KW - Child Offender KW - Child Violence KW - Juvenile Aggression KW - Juvenile Offender KW - Juvenile Violence KW - Exposure to Violence Language: en

Deliberative democrats aim to reform real-world politics. This goal, however, poses a dilemma for researchers assessing deliberation. Researchers can study deliberation through controlled but artificial laboratory and survey experiments. They can confidently infer causality (high internal validity) but cannot be sure that deliberation will operate the same way in applied politics (low external validity). Meanwhile, researchers can observe naturally occurring deliberation in realistic settings. In this case, they can be confident that the patterns they find are politically relevant (high external validity) but cannot be sure that they represent causal relationships (low internal validity). Without causality, however, scholars of deliberation cannot know whether reforms—changes to the system—will work as intended. Deliberative field experiments can help resolve this dilemma by conducting real experiments on real politics (achieving high internal and external validity). Like any research method, though, field experiments have their pitfalls, and so they are no panacea. This chapter explains the concept and mechanics of deliberative field experiments, links them to broader theories of deliberative democracy, and characterizes the kinds of questions that they are well suited to answering. The chapter concludes by discussing some of the limitations of deliberative field experiments and how they can be combined with other research methods to assess the promise and perils of deliberative reform more thoroughly.

BackgroundConjoint Analysis (CA) can serve as an important tool to study health disparities and unique factors underlying decision-making in diverse subgroups. However, methodological advancements are needed in exploiting this application of CA. We compared the internal and external predictive validity and inter-temporal stability of Choice-based-Conjoint (CBC) analysis between African-Americans and Whites in the clinical context of preferences for analgesic treatment for cancer pain.MethodsWe conducted a prospective study with repeated-measures at two time-points (T1 = baseline; T2 = 3-months). African-Americans (n = 102); and Whites (n = 139) with cancer-related pain were recruited from outpatient oncology clinics in Philadelphia. Informed by pilot work, a computer-assisted CBC experiment was developed using 5 attributes of analgesic treatment: type of analgesic; expected pain relief; type of side-effects; severity of side-effects; and out-of-pocket cost. The design included 2 choice alternatives, 12 random tasks, 2 holdout tasks, and maximum of 6 levels per attribute. The internal and external predictive validity of CBC was estimated using Root Likelihood (RLH) and Mean Absolute Error (MAE), respectively. Inter-temporal stability was assessed using Cohen’s kappa.ResultsWhites predominantly traded based on “pain relief” whereas African-Americans traded based on “type of side-effects”. At both time-points, the internal validity (RLH) was slightly higher for Whites than for African-Americans. The RLH for African-Americans improved at T2, possibly due to the learning effect. Lexicographic (dominant) behavior was observed in 29% of choice datasets; Whites were more likely than African-Americans to engage in a lexicographic behavior (60% vs. 40%). External validity (MAE) was slightly better for African-Americans than for Whites at both time-points (MAE: T1 = 3.04% for African-Americans and 4.02% for Whites; T2 = 8.04% for African-Americans; 10.24% for Whites). At T2, the MAE increased for both groups possibly reflecting an increase in the complexity of pain treatment decision-making based on expectations (T1) as opposed to reality (T2). The inter-temporal stability was fair for CBC attributes between T1 and T2 (kappa = 0.28, 95% CI: 0.24-0.32) and was not predicted by demographics including race.ConclusionsWhile we found slight group differences, overall the internal and external predictive validity of CBC was comparable between African-Americans and Whites. We discuss some areas to investigate and improve internal and external predictive validity of CBC experiments.

Purpose: The purpose of this study is twofold. First, we propose a typology of quantitative empirical research in management accounting based on two design features: presence of control group and sample representativeness. Second, we discuss implications of the methods for trade-offs between internal and external validity.
 Method: Based on previous methodological studies we develop a typology with eight quantitative empirical methods.
 Results: Based on the two design features, we propose eight quantitative empirical methods for management accounting studies: (1) laboratory experiment, (2) crowdsourcing experiment, (3) field experiment, (4) natural experiment, (5) single entity survey, (6) proprietary archival study, (7) large-scale survey and (8) pre-structured archival study. In addition, we critically compare the trade-offs and discuss the implications of these methods for internal and external validity.
 Contributions: The contribution is twofold. First, the proposed typology can help junior management accounting researchers increase the familiarity with the available empirical methods, some of which are still incipient in Brazil. Second, this study states that the choice of an empirical method typically implies benefits in terms of a validity type (e.g. internal validity) at the expense of other validity type (e.g., external validity). Claims of causality and results generalizability depend on which validity type is prioritized and remedies adopted to increase the overall validity of a study’s results.

Anatomical and clinical characteristics to guide decision making between coronary artery bypass surgery and percutaneous coronary intervention for individual patients: development and validation of SYNTAX score II

The evidence pyramid and introduction to randomized controlled trials

BackgroundGastrointestinal bleeding (GIB) is a common and often serious complication after stroke. Although several risk factors for post-stroke GIB have been identified, no reliable or validated scoring system is currently available to predict GIB after acute stroke in routine clinical practice or clinical trials. In the present study, we aimed to develop and validate a risk model (acute ischemic stroke associated gastrointestinal bleeding score, the AIS-GIB score) to predict in-hospital GIB after acute ischemic stroke.MethodsThe AIS-GIB score was developed from data in the China National Stroke Registry (CNSR). Eligible patients in the CNSR were randomly divided into derivation (60%) and internal validation (40%) cohorts. External validation was performed using data from the prospective Chinese Intracranial Atherosclerosis Study (CICAS). Independent predictors of in-hospital GIB were obtained using multivariable logistic regression in the derivation cohort, and β-coefficients were used to generate point scoring system for the AIS-GIB. The area under the receiver operating characteristic curve (AUROC) and the Hosmer-Lemeshow goodness-of-fit test were used to assess model discrimination and calibration, respectively.ResultsA total of 8,820, 5,882, and 2,938 patients were enrolled in the derivation, internal validation and external validation cohorts. The overall in-hospital GIB after AIS was 2.6%, 2.3%, and 1.5% in the derivation, internal, and external validation cohort, respectively. An 18-point AIS-GIB score was developed from the set of independent predictors of GIB including age, gender, history of hypertension, hepatic cirrhosis, peptic ulcer or previous GIB, pre-stroke dependence, admission National Institutes of Health stroke scale score, Glasgow Coma Scale score and stroke subtype (Oxfordshire). The AIS-GIB score showed good discrimination in the derivation (0.79; 95% CI, 0.764-0.825), internal (0.78; 95% CI, 0.74-0.82) and external (0.76; 95% CI, 0.71-0.82) validation cohorts. The AIS-GIB score was well calibrated in the derivation (P = 0.42), internal (P = 0.45) and external (P = 0.86) validation cohorts.ConclusionThe AIS-GIB score is a valid clinical grading scale to predict in-hospital GIB after AIS. Further studies on the effect of the AIS-GIB score on reducing GIB and improving outcome after AIS are warranted.

In experimental research must have external and internal validity in order to control secondary variables. External validity relates to the generalization of research results, while internal validity relates to the extent of the causal relationship between the independent variable and the dependent variable. Therefore, it is important to examine internal and external validity in experimental design. The method used in this study is a systematic literature review. The database used for literature search utilizes two digital libraries, namely Google Scholar and Scopus. The literature search will take place in May 2024. In order for the selected literature to be relevant, the search uses the keyword "internal external validity". The analysis technique used is a narrative method by grouping the data being explored. This research is expected to provide information related to internal and external validity in experimental design

The “Mathews Committee” (Mathews, 1990) concluded that the research performance of the accounting discipline in Australia was weak. This paper is motivated by the need to improve the quality of accounting research in Australia and probably some other countries as well. It discusses three issues of crucial importance to research quality ‐ internal, external, and construct validity with the discussion illustrated with examples from existing accounting research. Internal validity is concerned with the ability to make causal statements from a piece of research. External validity is concerned with the ability to generalize from the research and construct validity is interested in the validity of the variables measured. Internal and external validity are discussed within the context of five common research designs. No one design is strong in both types of validity and tradeoffs between internal and external validity are necessary. When designing a new research project, the particular tradeoffs made should depend on the research objectives. The paper discusses how these choices might be made and how, if internal validity problems exist, they might be minimized. With construct validity, no tradeoffs are appropriate ‐ the researcher should attempt to eliminate this type of invalidity. The paper discusses how this might be done.

The authors discuss criteria for the validation of psychoanalytic theories and develop a heuristic and normative model of the references needed for this. Their core question in this paper is: can psychoanalytic theories be validated exclusively from within psychoanalytic theory (internal validation), or are references to sources of knowledge other than psychoanalysis also necessary (external validation)? They discuss aspects of the classic truth criteria correspondence and coherence, both from the point of view of contemporary psychoanalysis and of contemporary philosophy of science. The authors present arguments for both external and internal validation. Internal validation has to deal with the problems of subjectivity of observations and circularity of reasoning, external validation with the problem of relevance. They recommend a critical attitude towards psychoanalytic theories, which, by carefully scrutinizing weak points and invalidating observations in the theories, reduces the risk of wishful thinking. The authors conclude by sketching a heuristic model of validation. This model combines correspondence and coherence with internal and external validation into a four-leaf model for references for the process of validating psychoanalytic theories.

The authors discuss criteria for the validation of psychoanalytic theories and develop a heuristic and normative model of the references needed for this. Their core question in this paper is: can psychoanalytic theories be validated exclusively from within psychoanalytic theory (internal validation), or are references to sources of knowledge other than psychoanalysis also necessary (external validation)? They discuss aspects of the classic truth criteria correspondence and coherence, both from the point of view of contemporary psychoanalysis and of contemporary philosophy of science. The authors present arguments for both external and internal validation. Internal validation has to deal with the problems of subjectivity of observations and circularity of reasoning, external validation with the problem of relevance. They recommend a critical attitude towards psychoanalytic theories, which, by carefully scrutinizing weak points and invalidating observations in the theories, reduces the risk of wishful thinking. The authors conclude by sketching a heuristic model of validation. This model combines correspondence and coherence with internal and external validation into a four‐leaf model for references for the process of validating psychoanalytic theories.

The author certifies that he has no commercial associations that might pose a conflict of interest in connection with the submitted article.