Terms Of Internal Validity Research Articles

Revisiting an Analysis of Threats to Internal Validity in Multiple Baseline Designs

In our previous article on threats to internal validity of multiple baseline design variations (Slocum et al., 2022), we argued that nonconcurrent multiple baseline designs (NCMB) are capable of rigorously demonstrating experimental control and should be considered equivalent to concurrent multiple baselines (CMB) in terms of internal validity. We were fortunate to receive five excellent commentaries on our article from experts in single-subject research design—four of whom endorsed the conclusion that NCMBs should be considered strong experimental designs capable of demonstrating experimental control. In the current article, we address the most salient points made in the five commentaries by further elaborating and clarifying the logic described in our original article. We address arguments related to classic threats including maturation, testing and session experience, and coincidental events (history). We rebut the notion that although NCMBs are strong, CMBs provide an increment of additional control and discuss the application of probability-based analysis of the likelihood of threats to internal validity. We conclude by emphasizing our agreement with many of the commentaries that selection of single-case experimental designs should be based on the myriad subtleties of research priorities and contextual factors rather than on a decontextualized hierarchy of designs.

The Development of a Rapid Classification Scale for Sleep Quality in Community-Dwelling Older Adults - The Yilan Study, Taiwan.

ObjectivePoor sleep quality is prevailing, deleterious, but heterogeneous in older adults. This study aimed to develop a simplified instrument to screen and classify poor sleep quality in community-dwelling older adults, by which stepped care with needs-based interventions could be implemented.MethodsCohorts of adults aged 65 years and older were used to develop the Rapid Classification Scale for Sleep Quality (RCSSQ). Poor sleep quality was defined with the Pittsburgh Sleep Quality Index (PSQI). Established subgroups of poor sleep quality in the development dataset (n = 2622) were used as the criterion standard. Two independent validation datasets (n = 964 and 193, respectively) were used to examine the external validity. Questions in the PSQI were examined by the stepwise multinomial logistic regressions to determine the optimal numbers of items in the RCSSQ. On the premise of item parsimony and instrument validity, the optimal combination of reduced items was determined.ResultsIn the development dataset, the 4-item RCSSQ (RCSSQ-4) was the optimal predictive model. In terms of internal validity, the accuracy rates to identify PSQI-defined poor sleep quality and its subgroups in the developmental dataset by the RCSSQ-4 were 89.0% and 79.9%, respectively. Meanwhile, the RCSSQ-4 also had good external validity in the validation datasets to detect PSQI-defined poor sleep quality (accuracy rates: 89.1–90.7%). Furthermore, the profiles of PSQI component scores and comorbid conditions for the predicted subgroups in the validation dataset were comparable with the criterion standard.ConclusionThe RCSSQ-4 is a valid instrument for screening and subgrouping poor sleep quality in community-dwelling older adults. The RCSSQ-4 may help guide tailored interventions under the context of stepped care in the community.

A Causal Mechanistic Perspective on Micro-Foundations in Family Business: The Case of CE

This paper is a conceptual contribution to the debate on micro- foundations in family business research. We advocate a causal- mechanistic approach to micro-foundations which is centered on the notion of causal mechanisms as responsible for the emergence of phenomena at the organizational level, such as corporate entrepreneurship in family firms. We distinguish two alternative purposes of the research process that can take advantage from it: prediction and explanation. First, we argue that knowledge of causal mechanisms can be exploited when testing hypothesis that draw on the mechanistic potential of dominant theories in family business research thus leading to better predictions. Second, we show how the mechanistic reasoning is a natural ally of major forms of explanation in the field thus improving current explanatory accounts in terms of internal validity, completeness and accuracy. The paper concludes by summarizing the epistemic benefits of the causal-mechanistic approach and offering suggestions for future research.

Transfer of arthroscopic skills from computer simulation training to the operating theatre: a review of evidence from two randomised controlled studies

Introduction: There is paucity in the research on transfer validity of arthroscopic simulator training. The aim of this article is to determine whether skills derived from arthroscopic simulation are transferrable to the operating theatre and retained over time. Methods: A systematic review with rigorous criteria to identify the highest level of evidence available was carried out. The studies were critically appraised with narrative data synthesis. Results: Twenty-one studies on arthroscopic simulation were identified. Only two studies were randomised controlled trials. The first article demonstrated improved performance of basic knee arthroscopic tasks following a fixed period of training. The second article showed improved performance of arthroscopic tasks and no deterioration in the levels of skills following a period of six months. In addition, the two studies succeeded in demonstrating the importance of 3D motion analysis using computer simulators in the assessment of technical skills. Components of evaluation such as time to task completion, distance travelled by instruments and incidence of instruments collisions were associated with the highest validity and reliability of assessment. This systematic review highlighted the limitations of these two randomised studies. Discussion: Evidence from the two trials suggests that knee arthroscopy simulator training can result in improved performance. This review helped highlight the contribution of the two studies in terms of internal validity and consistency of using arthroscopic skills training. Further level I studies are however required to demonstrate the evidence for transfer and predictive validity of computer simulation as a training instrument.

Strengthening nonrandomized studies of health communication strategies for HIV prevention.

There is growing interest in impact evaluations of health communication (HC) interventions for HIV prevention. Although cluster randomized trials may be optimal in terms of internal validity, they are often unfeasible for political, practical, or ethical reasons. However, a common alternative, the observational study of individuals who do and do not self-report HC intervention exposure, is prone to bias by confounding. Cluster-level quasi-experimental study designs offer promising alternatives to these extremes. We identified common rollout strategies for HC initiatives. We mapped these scenarios against established quasi-experimental evaluation designs. We identified key issues for implementers and evaluators if these designs are to be more frequently adopted in HC intervention evaluations with high internal validity. Stronger evaluations will document the planned intervention components in advance of delivery and will implement interventions in clusters according to a predefined systematic allocation plan. We identify 4 types of allocation plan and their associated designs. Where some places get the HC intervention, whereas others do not, a nonrandomized controlled study may be feasible. Where HC is introduced everywhere at a defined point in time, an interrupted time series may be appropriate. Where the HC intervention is introduced in phases, a nonrandomized phased implementation or stepped-wedge design may be used. Finally, where there is variation in strength of implementation of HC, a nonrandomized, dose-response study can be planned. Our framework will assist teams planning such evaluations by identifying critical decisions for the implementers and for the evaluators of HC interventions.

Assessing the appropriateness of the level of care for morbidly obese subjects: validation of the CASCO-R scale.

Aim of this study was to validate the Comprehensive Appropriateness Scale for the Care of Obesity in Rehabilitation (CASCO-R) and to determine the cut-off score for indicating the most appropriate health care setting for patients with obesity. The CASCO-R scale was developed according to the available scientific literature and expertise of an expert panel working for a Consensus document endorsed by the Italian Society of Obesity (SIO) and the Italian Society for the Study of Eating Disorders (SISDCA). 16 Italian centres, specialized in the treatment of obesity, characterised by different settings of care (specialist outpatient service, day-hospital service, intensive inpatient rehabilitation), participated in the study. 449 obese subjects were enrolled in the study (30.5% males and 69.5% females): 38.3% from outpatient services, 20.7% from day-hospital services and 40.9% from intensive inpatient rehabilitation units. After 2-month of treatment, a workload summary sheet, including medical and nursing interventions, number of expert advices and diagnostic procedures, and adverse clinical events (ACEs) was fulfilled for each patient. Statistically significant correlation was found between the CASCO-R scale score, overall workload and ACEs. The CASCO-R scale demonstrated also an excellent performance in terms of internal validity and test-retest analysis. Three total score cut-off have been proposed: >25 for inpatient intensive rehabilitation; 20-25 for day-hospital service; <20 for outpatient treatment. In conclusion, the CASCO-R scale was demonstrated to be a valid tool for assessing the appropriateness of the choice of the level of care. Hence, it can be used to verify the proper allocation of patients, as it was well correlated with measures of workload and the incidence of ACEs.

Do health economic evaluations using observational data provide reliable assessment of treatment effects?

Economic evaluation in modern health care systems is seen as a transparent scientific framework that can be used to advance progress towards improvements in population health at the best possible value. Despite the perceived superiority that trial-based studies have in terms of internal validity, economic evaluations often employ observational data. In this review, the interface between econometrics and economic evaluation is explored, with emphasis placed on highlighting methodological issues relating to the evaluation of cost-effectiveness within a bivariate framework. Studies that satisfied the eligibility criteria exemplified the use of matching, regression analysis, propensity scores, instrumental variables, as well as difference-in-differences approaches. All studies were reviewed and critically appraised using a structured template. The findings suggest that although state-of-the-art econometric methods have the potential to provide evidence on the causal effects of clinical and policy interventions, their application in economic evaluation is subject to a number of limitations. These range from no credible assessment of key assumptions and scarce evidence regarding the relative performance of different methods, to lack of reporting of important study elements, such as a summary outcome measure and its associated sampling uncertainty. Further research is required to better understand the ways in which observational data should be analysed in the context of the economic evaluation framework.

Adaptability: Conceptual and Empirical Perspectives on Responses to Change, Novelty and Uncertainty

Adaptability is proposed as individuals’ capacity to constructively regulate psycho-behavioral functions in response to new, changing, and/or uncertain circumstances, conditions and situations. The present investigation explored the internal and external validity of an hypothesised adaptability scale. The sample comprised 2,731 high school students. In terms of internal validity, exploratory and confirmatory factor analysis (EFA and CFA) suggested a reliable higher order adaptability factor subsumed by a reliable first order cognitive-behavioural factor and a reliable first order affective factor. Multi-group CFA indicated invariance in factor structure as a function of gender, age, and language background. Further, age (younger adolescents), language background (non-English speaking), and parents’ education (higher levels) predicted higher order adaptability, while gender (males) predicted first order affective adaptability. In terms of external validity, consistent with hypotheses, higher and first order adaptability was differentially associated with cognate/aligned factors (personality, implicit theories of ability, buoyancy) and also with psycho-educational wellbeing ‘outcome’ factors (achievement, enjoyment of school, meaning and purpose, life satisfaction). Findings hold theoretical and empirical implications for researchers and practitioners seeking to better understand the constructive regulation of individuals confronted with situations involving novelty, change, and uncertainty.

CMV Infection after Transplant from Cord Blood Compared to Other Alternative Donors: The Importance of Donor-Negative CMV Serostatus

Cytomegalovirus (CMV) infection and disease are important complications after hematopoietic stem cell transplant, particularly after transplant from alternative donors. Allogeneic cord blood transplantation (CBT) is being increasingly used, but immune recovery may be delayed. The aim of this study was to compare CMV infection in CBT with transplants from unrelated or mismatched related donors, from now on defined as alternative donors. A total of 165 consecutive transplants were divided in 2 groups: (1) alternative donors transplants (n = 85) and (2) CBT recipients (n = 80). Donor and recipient (D/R) CMV serostatus were recorded. The incidence of CMV infection, its severity, timing, and outcome were compared. Median follow-up was 257 days (1-1328). CMV infection was monitored by CMV antigenemia and expressed as CMV Ag positive cell/2 × 10(5) polymorphonuclear blood cells. There was a trend toward a higher cumulative incidence of CMV infection among CBT than alternative donor transplant recipients (64% vs 51%, P = .12). The median time to CMV reactivation was 35 days, and was comparable in the 2 groups (P = .8). The maximum number of CMV-positive cells was similar in the 2 groups (11 versus 16, P = .2). The time interval between the first and the last positive CMV antigenemia was almost 4 times longer in CBT compared with alternative donor transplants (109 vs 29 days, respectively, P = .008). The incidence of late CMV infection was also higher in CBT (62% vs 24%, P < .001). The incidence of early and late CMV infection in CBT was similar to D-/R+ alternative transplants, and higher than in D+/R+ alternative transplants: early infection, 72% in CBT versus 69% in D-/R+ alternative versus 55% in D+/R+ alternative (P = .21); and late infection, 67% in CBT versus 60% in D-/R+ alternative versus 7% in D+/R+ alternative (P < .001). Transplant-related mortality and overall survival were similar between the groups: 34% versus 36% (P = .6) and 54% versus 46% (P = .3) for alternative transplant and CBT, respectively. Longer duration and higher incidence of late CMV infection was seen in CBT patients, when compared with alternative donor transplants, whereas no difference in mortality was observed. The duration and incidence of late CMV infection were similar when D-/R+ CBT were compared with D-/R+ alternative donor transplants.

Reliability, Validity, and Health Issues Arising From Questionnaires Used to Measure Psychosocial and Organizational Work Factors (POWFs) Among Hospital Nurses: A Critical Review

This systematic review assesses the validity of epidemiological questionnaires used to measure psychosocial and organizational work factors (POWFs) in nurses. Of the 632 articles published between 1980 and July 2008 identified in this review, 108 provide some data concerning analysis of the intrinsic characteristics of such instruments (content validity or conceptual basis, reliability, validation of internal construction) and their external validity with respect to health aspects (concurrent validity and predictive validity). Psychometric properties of generalist questionnaires validated among blue collar or white collar workers were also assessed in the nurse population. The Job Content Questionnaire (JCQ), because of its longevity and reputation, was the generalist questionnaire most used among this population. Although its structure often raises questions in the nurse population, its dimensions (mainly the control one) have been shown to be predictive of some health outcomes measured with "objective" indicators concerning absenteeism, injuries, and musculoskeletal disorders. Effort Reward Imbalance (ERI), which has a structure more stable among the nurse population, has shown concurrent validity in terms of intent to leave the nursing profession. No questionnaire specifically designed for nurses can claim to satisfy all of the recommendations in terms of internal validity. Nevertheless, the Practice Environment Scale-Nursing Work Index (PES-NWI) seems to be one of the most promising instruments because of its appropriateness (content validity), its structure, which has a rather good fit (construct validity), its ability to discriminate magnet hospitals like other NWI derivates (discriminant validity), and it has also been associated in cross-sectional studies with health outcomes, especially nurses' self-assessed mental health but also with patients' health outcomes objectively assessed (concurrent validity). However, elements for predictive validity are still lacking with NWI derivates. The Discussion provides recommendations for measuring POWFs, encompassing the use of external validated measurements.

Standard Cost Lists for Healthcare in Canada

A standard cost list is a listing of recommended costs for a selected group of services. Standard costs are used in economic evaluation studies to eliminate that proportion of cost differences between interventions that are due to cost differences between providers. In this article we provide a summary of cost lists for pharmaceutical economic evaluation purposes which have been developed in 2 provinces in Canada-Alberta and Manitoba. We then assess these 2 lists from 2 different viewpoints. First, we developed criteria for the internal and external validity of costs and, in light of these validity criteria, we assessed how the 2 standard cost lists compared with the 'ideal' measure of long run marginal costs. Second, we identified the criteria for the inter-provincial consolidation of standard cost measures (in order to develop a single, consolidated cost list); in light of these criteria, we assessed whether the degree to which the 2 separate lists could be consolidated. The lists achieved a considerable degree of external validity, but fared less well in terms of internal validity. However, these results depend on the 'ideal' measure of cost which is used. The lists, in the forms which were developed, are not easily consolidated into a single list. Further refined cost data would be needed in order to achieve consolidation.

Adaptive conjoint analysis versus selfexplicated models: Some empirical results

Adaptive Conjoint Analysis ( aca) from Sawtooth Software is a very popular software package for collecting and analyzing conjoint data. This paper compares aca to the more traditional (pencil-and-paper), self-explicated model in terms of internal validity. Internal validity (on holdout profiles) is measured by product moment correlations, first-choice hit incidence, and correct rank position incidence. Surprisingly, we found that the self-explicated model outperformed aca in terms of internal validity.