Threats To External Validity Research Articles

Perceived context typicality and beliefs in the generalizability of management research findings

Despite growing calls for a greater internationalization of management research, the discipline still struggles with the challenge of integrating diverse national contexts. While recent decades have seen a change toward a more equitable treatment of all national contexts, the belief that research conducted outside the United States is less generalizable remains strong. In this research note, we explore the general perceptions of what is considered a “typical” study context by associating them with authors' variable tendencies to report threats to external validity. Using a sample of 400 papers from seven top-tier management journals, we find that research based on non-US data tends to report more external validity threats, which makes it appear less generalizable. While the belief that the US constitutes a “typical” study context is shared by both US and non-US author teams, non-US co-authors tend to exhibit a relatively stronger bias against the generalizability of non-US samples in their studies. Collectively, our results contribute to the literature on external validity threats, generalizability, and biases in peer review, while also responding to recent calls for a more diverse and inclusive management research program.

Methods in causal inference. Part 3: measurement error and external validity threats

Abstract The human sciences should seek generalisations wherever possible. For ethical and scientific reasons, it is desirable to sample more broadly than ‘Western, educated, industrialised, rich, and democratic’ (WEIRD) societies. However, restricting the target population is sometimes necessary; for example, young children should not be recruited for studies on elderly care. Under which conditions is unrestricted sampling desirable or undesirable? Here, we use causal diagrams to clarify the structural features of measurement error bias and target population restriction bias (or ‘selection restriction’), focusing on threats to valid causal inference that arise in comparative cultural research. We define any study exhibiting such biases, or confounding biases, as weird (wrongly estimated inferences owing to inappropriate restriction and distortion). We explain why statistical tests such as configural, metric and scalar invariance cannot address the structural biases of weird studies. Overall, we examine how the workflows for causal inference provide the necessary preflight checklists for ambitious, effective and safe comparative cultural research.

Agile Effort Estimation: Have We Solved the Problem Yet? Insights From a Replication Study

In the last decade, several studies have explored automated techniques to estimate the effort of agile software development. We perform a close replication and extension of a seminal work proposing the use of Deep Learning for Agile Effort Estimation (namely Deep-SE), which has set the state-of-the-art since. Specifically, we replicate three of the original research questions aiming at investigating the effectiveness of Deep-SE for both within-project and cross-project effort estimation. We benchmark Deep-SE against three baselines (i.e., Random, Mean and Median effort estimators) and a previously proposed method to estimate agile software project development effort (dubbed TF/IDF-SVM), as done in the original study. To this end, we use the data from the original study and an additional dataset of 31,960 issues mined from TAWOS, as using more data allows us to strengthen the confidence in the results, and to further mitigate external validity threats. The results of our replication show that Deep-SE outperforms the Median baseline estimator and TF/IDF-SVM in only very few cases with statistical significance (8/42 and 9/32 cases, respectively), thus confounding previous findings on the efficacy of Deep-SE. The two additional RQs revealed that neither augmenting the training set nor pre-training Deep-SE play lead to an improvement of its accuracy and convergence speed. These results suggest that using semantic similarity is not enough to differentiate user stories with respect to their story points; thus, future work has yet to explore and find new techniques and features that obtain accurate agile software development estimates.

The Role of Intelligent Technologies in Early Detection of Autism Spectrum Disorder (ASD): A Scoping Review

Background: Two-year delay is reported between the first developmental concern raised by the parents and the diagnosis of ASD (Autism Spectrum Disorder), delaying the start of early intervention programs most beneficial within the first three years. Aim: Evaluate the role of technology in ASD detection by answering four research questions analyzing 1) evolution of technology, 2) use of various bio-behavioral data sources, 3) demographic categories, databases, controls, comparators, and assessment instruments, and 4) data collection, processing, and outcomes of the technology-based methods in ASD detection. Methods: Scoping review included behavioral-based ASD screening and diagnostic studies, published between 1st January 2011 to 31st December 2021 in PUBMED, SCOPUS, and IEEE Xplore databases for children under six years. The studies were assessed using the Critical Appraisal Skills Programm (CASP) and the PRISMA scoping review checklist (PRISMA-ScR). Results: The shortlisted 35 studies were categorized into seven bio-behavioral categories. The review suggested extensive use of machine learning (ML) and Deep Learning (DL) technologies with multimodal structured and unstructured data to detect infants at risk of ASD and Other developmental delays (ODD) as early as 9 to 12 months. However, the review reported various internal and external validity threats. Conclusion: Technology can significantly improve the current ASD detection process. The validation and adoption of technology can be fast-tracked by 1) designing robust study protocols, 2) executing multi-cultural field trials, 3) standardizing datasets, data quality and feature engineering methods, 4) recruiting statistically significant participants from ASD, typically developing (TD) and other developmental disorders (ODD) groups to ensure technological generalization, validation, and adoption outside laboratory settings.

The effectiveness of intervention for adolescents exposed to domestic violence

The purpose of this study is to evaluate the effectiveness of a 10-week group intervention for adolescents who were exposed to domestic violence in their home, and compare behavioral and emotional symptoms between adolescents with direct or indirect exposure to domestic violence. Participants of the group intervention (N=32) reported an increase in knowledge of domestic violence based on quiz scores of pre- and post-tests. Compared to those without direct exposure to domestic violence, the participants that were directly exposed to domestic violence presented greater symptoms in the subscales of the Child Behavioral Checklist and the Revised Children's Anxiety and Depression Scale. Such findings are discussed along with previous literature. The study also discusses limitations related to internal and external validity threats and suggestions for future replication of the group intervention toward adolescents who were exposed to domestic violence and the related program evaluation.

Design and Reporting of Targeted Anticancer Preclinical Studies: A Meta-Analysis of Animal Studies Investigating Sorafenib Antitumor Efficacy.

The validity of preclinical studies of candidate therapeutic agents has been questioned given their limited ability to predict their fate in clinical development, including due to design flaws and reporting bias. In this study, we examined this issue in depth by conducting a meta-analysis of animal studies investigating the efficacy of the clinically approved kinase inhibitor, sorafenib. MEDLINE, Embase, and BIOSIS databases were searched for all animal experiments testing tumor volume response to sorafenib monotherapy in any cancer published until April 20, 2012. We estimated effect sizes from experiments assessing changes in tumor volume and conducted subgroup analyses based on prespecified experimental design elements associated with internal, construct, and external validity. The meta-analysis included 97 experiments involving 1,761 animals. We excluded 94 experiments due to inadequate reporting of data. Design elements aimed at reducing internal validity threats were implemented only sporadically, with 66% reporting animal attrition and none reporting blinded outcome assessment or concealed allocation. Anticancer activity against various malignancies was typically tested in only a small number of model systems. Effect sizes were significantly smaller when sorafenib was tested against either a different active agent or combination arm. Trim and fill suggested a 37% overestimation of effect sizes across all malignancies due to publication bias. We detected a moderate dose-response in one clinically approved indication, hepatocellular carcinoma, but not in another approved malignancy, renal cell carcinoma, or when data were pooled across all malignancies tested. In support of other reports, we found that few preclinical cancer studies addressed important internal, construct, and external validity threats, limiting their clinical generalizability. Our findings reinforce the need to improve guidelines for the design and reporting of preclinical cancer studies. Cancer Res; 76(16); 4627-36. ©2016 AACR.

Investigating the Effects of Sensory Integration Therapy in Decreasing Stereotypy

Sensory Integration Therapy (SIT) is a popular treatment for Pervasive Developmental Disorders that involves therapists using various strategies and manipulanda to provide sensory stimulation to improve behavioral dysfunctions. Although SIT is popular, the research literature demonstrates little experimental proof of effectiveness. Many published studies find little to no causal relationship between SIT and improvements in target behaviors. There are numerous internal and external validity threats that preclude confidence in a functional relationship for those studies that report positive changes. The current study attempted to evaluate the impact of different SIT techniques on the behavioral excesses of children diagnosed with autism, while using research designs that adhered to commonly accepted standards for internal and external validity controls. The results showed that there was no causal relationship between the sensory procedures and improvements in the targeted dependent variables. Thus, SIT remains an unproven treatment for autism.

The perils of ignoring design effects in experimental studies: Lessons from a mammography screening trial

Objective : Threats to external validity, including pretest sensitisation and the interaction of selection and an intervention, are frequently overlooked by researchers despite their potential to significantly influence study outcomes. The purpose of this investigation was to conduct secondary data analyses to assess the presence of external validity threats in the setting of a randomised trial designed to promote mammography use in a high-risk sample of women. Design : During the trial, recruitment and intervention, implementation took place in three cohorts (with different ethnic composition), utilising two different designs (pretest–posttest control group design and posttest only control group design). Results : Results reveal that the intervention produced different outcomes across cohorts, dependent upon the research design used and the characteristics of the sample. Conclusion : These results illustrate the importance of weighing the pros and cons of potential research designs before making a selection and attending more closely to issues of external validity.

Internal audit involvement in enterprise risk management

PurposeThe purpose of this paper is to examine the impact of internal auditors' involvement in enterprise risk management (ERM) on perceptions of their willingness to report a breakdown in risk procedures and whether a strong relationship with the audit committee affects such willingness to report. The study also investigates the use of ERM and the role of internal audit in ERM in Australian private and public sector entities.Design/methodology/approachThe study uses an experimental design, manipulating the internal auditor's involvement in ERM and the strength of the relationship between internal audit and the audit committee. Participants are 117 certified internal auditors. The study also gathers descriptive data on the use of ERM.FindingsThe study indicates that a high involvement in ERM impacts the perceptions of internal auditors' willingness to report a breakdown in risk procedures to the audit committee. However, a strong relationship with the audit committee does not appear to affect their perceived willingness to report. The study also finds that the majority of organisations have recently adopted ERM. Internal auditors are involved in ERM assurance activities but some also engage in activities that could compromise objectivity.Research limitations/implicationsThere are internal and external validity threats associated with the experimental design.Practical implicationsThe findings reinforce the need for organisations to adhere to the recommendations of the Institute of Internal Auditors and to ensure that internal auditors do not play an inappropriate role in ERM.Originality/valueThe paper contributes to our understanding of the impact of involvement in ERM on internal audit objectivity and of the current role of internal audit in ERM in Australia.

The clinical effectiveness of guided self-help versus waiting-list control in the management of anxiety and depression: a randomized controlled trial

There are significant barriers to accessing effective psychological therapy in primary care resulting from a lack of suitably trained therapists to meet current demand. More efficient service delivery using minimal interventions (such as bibliotherapy) provided by paraprofessional therapists may be one method of overcoming these problems, and is the subject of attention in the UK and elsewhere. A randomized trial was conducted to test the clinical effectiveness of this model. Assistant psychologists delivered a guided self-help intervention to patients with anxiety and depression who were currently waiting for psychological therapy. A total of 114 patients were randomized either to guided self-help or a waiting-list control group. All patients were followed up 3 months later, prior to starting conventional psychological therapy. Measures included self-reported adherence to the intervention, anxiety and depressive symptoms, social functioning and patient satisfaction. Adherence to the guided self-help intervention was acceptable and patients reported satisfaction with the intervention. However, there were no statistically significant differences between groups in anxiety and depression symptoms at 3 months. The results demonstrate that this model of guided self-help did not provide additional benefit to patients on a waiting list for psychological therapy. The results are considered in the context of possible internal and external validity threats, and compared with previous trials of minimal interventions. The implications of the results for the design of future minimal interventions are considered.

Panel Attrition and External Validity in the Short-Term Follow-Up Study of Adolescent Substance Use

The present study examines the relationship among grade, substance assessed; and type of measurement and statistical analysis employed in the detection of external validity threats due to panel attrition in the modal follow-up study found in adolescent substance use research. Six successive 2-year panel samples are examined The results indicate that dropouts demonstrate significantly higher baseline mean use of licit (tobacco and alcohol) and illicit (marijuana and hard drugs) substances when ANOVA is used, but only higher baseline use of illicit substances when chi-square analyses are used In addition, although mean use tests are no more sensitive to the increased variation in substance use that occurs during adolescence, they are more likely to reveal differences between dropouts and stayers in baseline use of licit substances. The results indicate (a) that findings from school-based, longitudinal studies almost certainly underestimate the level and extent ofsubstance use among adolescents; and (b) that alternative types of attrition analyses yield different effects regardless of grade, even in the short-term follow-up study. These findings are discussed for their implications to adolescent substance use research and for the establishment of standardized procedures to consider the relative risk of Type 2 to Type 1 error in attrition analyses.

A meta‐analytic review of the effect of sentence‐combining on reading comprehension

Abstract This study extended an earlier exploratory study (Searls & Neville, 1988) of the effects of sentence‐combining on reading achievement by updating the pool of studies and using meta‐analysis as a vehicle for integration. The original pool of 34 studies was reduced to 24 by applying criteria for germaneness, methodology, independence, and effect size anomalies. The 24 studies, 19 of which were dissertations, were examined as to control of internal and external validity threats. Effect sizes were used to compare the results of the studies on types of criterion measures, grade levels, control of validity threats, length of treatment, and recency of publication. Only grade levels produced significant differences in that sentence‐combining had a greater effect at the elementary level than at the high school or college levels. Conclusions related to sentence combining as an instructional technique and implications for further research using meta‐analysis are discussed.