Consideration Of External Validity Research Articles

Effects of chronic physical activity on cognition across the lifespan: a systematic meta-review of randomized controlled trials and realist synthesis of contextualized mechanisms

ABSTRACT In the surging field of chronic physical activity (PA) and cognition research, problems arise that prevent us from ‘seeing the forest for the trees’. The first aim was to identify them and propose solutions. Moreover, inconsistencies in conclusions of a rising amount of systematic reviews render necessary ‘an umbrella for a rain of evidence’. The second aim was to obtain a differentiated picture of moderators that may explain inconsistencies using the wide-angle lens of a systematic meta-review. We especially addressed the role of the PA context in causation mechanisms, complementing the meta-review with a realist approach that is best suited to identify context-mechanism-outcome (CMO) configurations. Main review outcomes are: (1) inconsistent grading of methodological quality and low consideration of external validity; (2) role of multiple moderators at participant-, intervention-, study-level but relative neglect of context-level moderators; (3) explanatory potential of CMO configurations, in which specific conditions of the context may trigger different mechanisms that generate PA effects on cognition. The majority of the proposed mechanisms converge on the concept of enrichment. Conclusions highlight the need for future research, in which PA interventions and their contexts are designed to mobilize the different mechanisms underpinning their individual and joint effects on cognition.

E-Mental Health Program Usage Patterns in Randomized Controlled Trials and in the General Public to Inform External Validity Considerations: Sample Groupings Using Cluster Analyses.

BackgroundRandomized controlled trials (RCTs) with vigorous study designs are vital for determining the efficacy of treatments. Despite the high internal validity attributed to RCTs, external validity concerns limit the generalizability of results to the general population. Bias can be introduced, for example, when study participants who self-select into a trial are more motivated to comply with study conditions than are other individuals. These external validity considerations extend to e-mental health (eMH) research, especially when eMH tools are designed for public access and provide minimal or no supervision.ObjectiveClustering techniques were employed to identify engagement profiles of RCT participants and community users of a self-guided eMH program. This exploratory approach inspected actual, not theorized, RCT participant and community user engagement patterns. Both samples had access to the eMH program over the same time period and received identical usage recommendations on the eMH program website. The aim of this study is to help gauge expectations of similarities and differences in usage behaviors of an eMH tool across evaluation and naturalistic contexts.MethodsAustralian adults signed up to myCompass, a self-guided online treatment program created to reduce mild to moderate symptoms of negative emotions. They did so either by being part of an RCT onboarding (160/231, 69.6% female) or by accessing the program freely on the internet (5563/8391, 66.30% female) between October 2011 and October 2012. During registration, RCT participants and community users provided basic demographic information. Usage metrics (number of logins, trackings, and learning activities) were recorded by the system.ResultsSamples at sign-up differed significantly in age (P=.003), with community users being on average 3 years older (mean 41.78, SD 13.64) than RCT participants (mean 38.79, SD 10.73). Furthermore, frequency of program use was higher for RCT participants on all usage metrics compared to community users through the first 49 days after registration (all P values <.001). Two-step cluster analyses revealed 3 user groups in the RCT sample (Nonstarters, 10-Timers, and 30+-Timers) and 2 user groups in the community samples (2-Timers and 20-Timers). Groups seemed comparable in patterns of use but differed in magnitude, with RCT participant usage groups showing more frequent engagement than community usage groups. Only the high-usage group among RCT participants approached myCompass usage recommendations.ConclusionsFindings suggested that external validity concerns of RCT designs may arise with regards to the predicted magnitude of eMH program use rather than overall usage styles. Following up RCT nonstarters may help provide unique insights into why individuals choose not to engage with an eMH program despite generally being willing to participate in an eMH evaluation study. Overestimating frequency of engagement with eMH tools may have theoretical implications and potentially impact economic considerations for plans to disseminate these tools to the general public.

Against external validity

Francesco Guala once wrote that ‘The problem of extrapolation (or external validity as it is sometimes called) is a minor scandal in the philosophy of science’. This paper agrees with the statement, but for reasons different from Guala’s. The scandal is not, or not any longer, that the problem has been ignored in the philosophy of science. The scandal is that framing the problem as one of external validity encourages poor evidential reasoning. The aim of this paper is to propose an alternative—an alternative which constitutes much better evidential reasoning about target systems of interest, and which makes do without (much) consideration of external validity.

Selecting and Improving Quasi-Experimental Designs in Effectiveness and Implementation Research.

Interventional researchers face many design challenges when assessing intervention implementation in real-world settings. Intervention implementation requires holding fast on internal validity needs while incorporating external validity considerations (such as uptake by diverse subpopulations, acceptability, cost, and sustainability). Quasi-experimental designs (QEDs) are increasingly employed to achieve a balance between internal and external validity. Although these designs are often referred to and summarized in terms of logistical benefits, there is still uncertainty about (a) selecting from among various QEDs and (b) developing strategies to strengthen the internal and external validity of QEDs. We focus here on commonly used QEDs (prepost designs with nonequivalent control groups, interrupted time series, and stepped-wedge designs) and discuss several variants that maximize internal and external validity at the design, execution and implementation, and analysis stages.

Systematic reviews need to consider applicability to disadvantaged populations: inter-rater agreement for a health equity plausibility algorithm

BackgroundSystematic reviews have been challenged to consider effects on disadvantaged groups. A priori specification of subgroup analyses is recommended to increase the credibility of these analyses. This study aimed to develop and assess inter-rater agreement for an algorithm for systematic review authors to predict whether differences in effect measures are likely for disadvantaged populations relative to advantaged populations (only relative effect measures were addressed).MethodsA health equity plausibility algorithm was developed using clinimetric methods with three items based on literature review, key informant interviews and methodology studies. The three items dealt with the plausibility of differences in relative effects across sex or socioeconomic status (SES) due to: 1) patient characteristics; 2) intervention delivery (i.e., implementation); and 3) comparators. Thirty-five respondents (consisting of clinicians, methodologists and research users) assessed the likelihood of differences across sex and SES for ten systematic reviews with these questions. We assessed inter-rater reliability using Fleiss multi-rater kappa.ResultsThe proportion agreement was 66% for patient characteristics (95% confidence interval: 61%-71%), 67% for intervention delivery (95% confidence interval: 62% to 72%) and 55% for the comparator (95% confidence interval: 50% to 60%). Inter-rater kappa, assessed with Fleiss kappa, ranged from 0 to 0.199, representing very low agreement beyond chance.ConclusionsUsers of systematic reviews rated that important differences in relative effects across sex and socioeconomic status were plausible for a range of individual and population-level interventions. However, there was very low inter-rater agreement for these assessments. There is an unmet need for discussion of plausibility of differential effects in systematic reviews. Increased consideration of external validity and applicability to different populations and settings is warranted in systematic reviews to meet this need.

Human subjects protection issues in QUERI implementation research: QUERI Series

BackgroundHuman Subjects protections approaches, specifically those relating to research review board oversight, vary throughout the world. While all are designed to protect participants involved in research, the structure and specifics of these institutional review boards (IRBs) can and do differ. This variation affects all types of research, particularly implementation research.MethodsIn 2001, we began a series of inter-related studies on implementing evidence-based collaborative care for depression in Veterans Health Administration primary care. We have submitted more than 100 IRB applications, amendments, and renewals, and in doing so, we have interacted with 13 VA and University IRBs across the United States (U.S.). We present four overarching IRB-related themes encountered throughout the implementation of our projects, and within each theme, identify key challenges and suggest approaches that have proved useful. Where applicable, we showcase process aids developed to assist in resolving a particular IRB challenge.ResultsThere are issues unique to implementation research, as this type of research may not fit within the traditional Human Subjects paradigm used to assess clinical trials. Risks in implementation research are generally related to breaches of confidentiality, rather than health risks associated with traditional clinical trials. The implementation-specific challenges discussed are: external validity considerations, Plan-Do-Study-Act cycles, risk-benefit issues, the multiple roles of researchers and subjects, and system-level unit of analysis.DiscussionSpecific aspects of implementation research interact with variations in knowledge, procedures, and regulatory interpretations across IRBs to affect the implementation and study of best methods to increase evidence-based practice. Through lack of unambiguous guidelines and local liability concerns, IRBs are often at risk of applying both variable and inappropriate or unnecessary standards to implementation research that are not consistent with the spirit of the Belmont Report (a summary of basic ethical principles identified by the National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research), and which impede the conduct of evidence-based quality improvement research. While there are promising developments in the IRB community, it is incumbent upon implementation researchers to interact with IRBs in a manner that assists appropriate risk-benefit determinations and helps prevent the process from having a negative impact on efforts to reduce the lag in implementing best practices.

The Congruence of Experimental Design with Statistical Analysis: Power and External-Validity Considerations from Multiple One-Way ANOVAs versus a Factorial Analysis

The Congruence of Experimental Design with Statistical Analysis: Power and External-Validity Considerations from Multiple One-Way ANOVAs versus a Factorial Analysis