Provide Evidence For Validity Research Articles

Initial development and validity evidence for the processes of change in Psychological Skills Training Questionnaire

ObjectivesThe purpose of this study was to create and provide validity evidence for the Processes of Change in Psychological Skills Training Questionnaire (PCPSTQ). DesignThe current study used a cross-sectional research design. MethodsFive hundred fifty nine NCAA Division I, professional, and Olympic level athletes participated in the current study. To create the PCPSTQ, an initial pool of 114 items was generated by the research team. After a content validity process, 65 items were retained for analysis. Exploratory structural equation modeling was used as an analytic strategy to identify the most appropriate factor structure for the PCPSTQ. Decisions about the most appropriate model were made using multiple fit indices. To examine the construct validity of the PCPSTQ, a series of one-way ANOVAs were conducted to examine differences in processes of change use across stage of change. ResultsIn the current study, validity evidence provided support for a 7-factor process of change measure (χ2 = 325.84, p < .001; Comparative Fit Index = .971; Tucker Lewis Index = .945; Root Mean Square Error of Approximation = .037; Standard Root Mean Square Residual = .020). Results also supported the construct validity of the scale as a significant difference in process of change use across stage of change was reported for all seven processes. ConclusionsResults of the current study support the factor structure and construct validity of the PCPSTQ. It appears that the processes of behavior change reported across multiple behavior change domains might also be viable for sport psychology professionals.

Much ado about differences: why expert-novice comparisons add little to the validity argument.

One approach to validating assessment scores involves evaluating the ability of scores to discriminate among groups who differ in a specific characteristic, such as training status (in education) or disease state (in clinical applications). Such known-groups comparison studies provide validity evidence of "relationships with other variables." The typical education research study might compare scores between staff physicians and postgraduate trainees with the hypothesis that those with more advanced training (the "experts") will have higher scores than those less advanced (the "novices"). However, such comparisons are too nonspecific to support clear conclusions, and expert-novice comparisons (and known-groups comparisons in general) thus contribute little to the validity argument. The major flaw is the problem of confounding: there are multiple plausible explanations for any observed between-group differences. The absence of hypothesized differences would suggest a serious flaw in the validity argument, but the confirmation of such differences adds little. As such, accurate known-groups discrimination may be necessary, but will never be sufficient, to support the validity of scores. This article elaborates on this and other problems with the known-groups comparison that limit its utility as a source of validity evidence.

Temperamental basis of sense of humor: The Spanish long form of the trait version of the State-Trait-Cheerfulness-Inventory

Despite the numerous definitions and dimensions proposed to explain the sense of humor and the variety of instruments developed for its assessment, little attention has been paid to its affective and attitudinal basis in the models developed so far. The long form of the trait version of the State-Trait-Cheerfulness-Inventory (STCI-T; Ruch, Köhler, & van Thriel, 1996) was developed using a facet approach to measure the temperamental basis of sense of humor using three theoretically-derived concepts: cheerfulness, seriousness, and bad mood. This paper presents the psychometric analysis of the Spanish long form of the trait version of the STCI-T. We assessed the dimensionality of the instrument, the internal consistency and test–retest reliability of its facets and scales and the relationships between STCI-T domains and other variables. We assessed four independent samples comprised of 1049 participants in total with ages ranging between 18 and 94years. The psychometric characteristics appeared to be satisfactory and proved to be replicable. Moreover, relationships between (a) the temperamental basis of sense of humor and (b) personality and well-being were also replicated. Results provide validity evidence for using the Spanish version of the STCI-T to assess the temperamental basis of sense of humor in the Spanish population.

Toward a Multidimensional Model of Athletes’ Commitment to Coach-Athlete Relationships and Interdependent Sport Teams: A Substantive-Methodological Synergy

Drawing from a three-factor model of organizational commitment, we sought to provide validity evidence for a multidimensional conceptualization designed to capture adolescent athletes' commitment to their coach-athlete relationship or their team. In Study 1, 335 individual-sport athletes (Mage = 17.32, SD = 1.38) completed instruments assessing affective, normative, and continuance commitment to their relationship with their coach, and in Study 2, contextually modified instruments were administered to assess interdependent-sport athletes' (N = 286, Mage = 16.31, SD = 1.33) commitment to their team. Bayesian structural equation modeling revealed support for a three-factor (in comparison with a single-factor) model, along with relations between commitment dimensions and relevant correlates (e.g., satisfaction, return intentions, cohesion) that were largely consistent with theory. Guided by recent advancements in Bayesian modeling, these studies provide a new commitment instrument with the potential for use and refinement in team- and relationship-based settings and offer preliminary support for a conceptual framework that may help advance our understanding of the factors underpinning individuals' engagement in sport.

Investigating Chinese Language Learners’ Reading Comprehension for Different Meaning Types

The Chinese government has published an official guide specifying the aims for reading in Chinese and the expected comprehension levels for different proficiency learners, with regard to teaching and learning Chinese as a second or foreign language. However, due to lack of teacher training for its implementation, this guide has rarely been used for teaching reading to Chinese language learners and has rarely been used for evaluating their reading ability. Therefore, it appears that teaching and assessing reading comprehension have not been based on a theoretical background of reading ability. The purpose of this study is twofold: (1) to provide validity evidence of a Chinese reading test developed based on a theoretical model of reading ability; and (2) to examine reading test performance of Chinese learners with various reading ability levels. For the purpose of this study, reading ability was defined based on a meaning-based model that included three layers of reading comprehension: literal, intended, and implied meanings of a reading text. A total of 248 Korean university students were divided into three levels, and their test performances were analyzed and compared for the three meaning types using structural equation modeling and regression analysis. The results suggest that the test performance structure represented the meaning-based model in general, thereby providing validity evidence of the test. Further analyses revealed that the three groups differed from one another with respect to their understanding of literal, intended, and implied meanings. The findings provide pedagogical implications for teaching Chinese language learners with different reading proficiency levels. DOI: http://dx.doi.org/10.17576/GEMA-2014-1401-06

Validating MMI scores: are we measuring multiple attributes?

The multiple mini-interview (MMI) used in health professional schools' admission processes is reported to assess multiple non-cognitive constructs such as ethical reasoning, oral communication, or problem evaluation. Though validation studies have been performed with total MMI scores, there is a paucity of information regarding how well MMI scores differentiate the constructs being measured, the relationship between MMI scores (construct or total) and personality characteristics, and how well MMI scores (construct or total) predict future performance in practice. Results from these studies could assist with MMI station development, rater training, score interpretation, and resource allocation. The purpose of this study was to investigate the validity of MMI construct scores (oral communication and problem evaluation), and their relationship to personality measures (emotionality and extraversion) and specific scores from standardized clinical communications interviews (building the relationship and explaining and planning). Confirmatory factor analysis results support a two factor MMI model, however the correlation between these factors was .87. Oral communication MMI scores significantly correlated with extraversion (r c = .25, p < .05), but MMI scores were not related to emotionality. Scores for building a relationship were significantly related to MMI oral communication scores, (r c = .46, p < .001) and problem evaluation scores (r c = .43, p < .001); scores for explaining and planning were significantly related to MMI problem evaluation scores (r c = .36, p < .01). The results provide validity evidence for assessing multiple non-cognitive attributes during the MMI process and reinforce the importance of developing MMI stations and scoring rubrics for attributes identified as important for future success in school and practice.

The teacher efficacy for inclusive practices (TEIP) scale: dimensionality and factor structure

The Teacher Efficacy for Inclusive Practices (TEIP) scale is designed to measure teacher‐self efficacy to teach in inclusive classrooms. The original study identified three scale factors: efficacy in using inclusive instruction (EII), efficacy in collaboration (EC), and efficacy in managing behavior (EMB) (Sharma et al., 2012). The purpose of our study was to examine the TEIP scale for dimensionality and to cross‐validate its factor structure for pre‐service teachers in the context of early childhood education. A bifactor model fit to the data revealed that the TEIP scale is essentially unidimensional, that is, there is one dominant latent factor and the originally found three scale factors (EII, EC, and EMB) represent specific aspects of the general factor of teacher self‐efficacy to teach in inclusive classrooms. Along with providing validation evidence, these findings have important implications for the scoring on the TEIP scale using classical test analysis or unidimensional item response theory models.

The Single-Item Math Anxiety Scale

This study examined whether the Single-Item Math Anxiety Scale (SIMA), based on the item suggested by Ashcraft, provided valid and reliable scores of mathematical anxiety. A large sample of university students ( n = 279) was administered the SIMA and the 25-item Shortened Math Anxiety Rating Scale (sMARS) to evaluate the relation between the scores of the two measures. The university students were also administered other tests to provide validity evidence for the SIMA scores. The temporal stability of the SIMA scores was also evaluated over a 7-week test-retest interval. The findings of the study demonstrated that the SIMA scores showed evidence of validity and strong test-retest reliability. We advocate for the use of the SIMA as a quick and useful means of assessing math anxiety, particularly in research and educational settings when large samples have to be assessed.

Assessing Residents’ Written Learning Goals and Goal Writing Skill

To provide validity evidence for use of the Learning Goal Scoring Rubric to assess the quality of written learning goals and residents' goal writing skills. This two-part study used the rubric to assess University of Colorado third-year pediatric residents' written learning goals to obtain validity evidence. In study 1, five raters independently scored 48 goals written in 2010-2011 and 2011-2012 by 48 residents, who also responded to the Jefferson Scale of Physician Lifelong Learning (JeffSPLL). In study 2, two raters independently scored 48 goals written in 2011-2012 by 12 residents. Intraclass correlation coefficients (ICCs) assessed rater agreement to provide evidence for response process. Generalizability theory assessed internal structure. Independent-samples Mann-Whitney U tests and correlations assessed relationship to other variables. Content was matched to published literature and instructional methods. The ICC was 0.71 for the overall rubric. In study 1, where the generalizability study's (G study's) object of measurement was learning goals, the phi coefficient was 0.867. In study 2, where the G study's object of measurement was the resident (goal writing skill), the phi coefficient was 0.751. The total mean score of residents with goal writing training was significantly higher than that of those without (7.54 versus 4.98, P < .001). Correlation between goal quality and JeffSPLL score was not significant. Investigators agreed that the content matched the published literature and instructional methods. Preliminary validity evidence indicates that this scoring rubric can assess learning goal quality and goal writing skill.

Explaining differences between retrospective and traditional pretest self-assessments: competing theories and empirical evidence

Summarizing theory and results of empirical research, this article serves to illustrate why effects measured with retrospective pretests may be subject to bias and may not always be explained by response shift theory. It presents three contending theories to explain the difference between retrospective and traditional pretest results and considers how the evaluation environment may inform subject bias. Four recommendations are made for workforce education (WE) researchers and practitioners who employ retrospective pretest data to report programme outcomes. WE professionals should (a) consider the cognitive implications of tasking participants to recall information, (b) select a robust evaluation design to encompass the retrospective pretest, (c) provide validity evidence of retrospective pretest data, and (d) conduct additional research to evaluate how elements of the evaluation process moderate retrospective assessments.

Development and validation of a self-efficacy scale for clinical decision-making in general paediatrics

Success in a task is not only related to skill, but it is also related to 'self-efficacy', or belief in one's capability to perform that task. No tool currently exists to measure self-efficacy in clinical decision-making in general paediatrics. To develop and provide validity evidence for the General Pediatrics-specific Self-Efficacy (GPedsSE) scale. The five-item GPedsSE scale, developed using an expert panel, was matched to the New General Self-Efficacy (NGSE) scale's structure for validity analysis purposes. Thirty-six postgraduate year 1 to postgraduate year 5 paediatric residents of the Montreal Children's Hospital (Montreal, Quebec) completed the GPedsSE and NGSE scales, with items interspersed. The mean (± SD) GPedsSE score was 18.6±2.6 of 25. Total GPedsSE and NGSE scores were moderately correlated (r=0.54, P<0.005). On planned comparison, the GPedsSE score increased with training year (F[1.3]=6.62; P<0.001), while the NGSE score did not (F<0.37). Exploratory factor analysis showed two components, each aligning with a scale. The GPedsSE scale contextualizes an existing tool to general paediatrics, a novel concurrent validity approach.

The Objective Assessment of Experts’ and Novices’ Suturing Skills Using An Image Analysis Program

To objectively assess suturing performance using an image analysis program and to provide validity evidence for this assessment method by comparing experts' and novices' performance. In 2009, the authors used an image analysis program to extract objective variables from digital images of suturing end products obtained during a previous study involving third-year medical students (novices) and surgical faculty and residents (experts). Variables included number of stitches, stitch length, total bite size, travel, stitch orientation, total bite-size-to-travel ratio, and symmetry across the incision ratio. The authors compared all variables between groups to detect significant differences and two variables (total bite-size-to-travel ratio and symmetry across the incision ratio) to ideal values. Five experts and 15 novices participated. Experts' and novices' performances differed significantly (P < .05) with large effect sizes attributable to experience (Cohen d > 0.8) for total bite size (P = .009, d = 1.5), travel (P = .045, d = 1.1), total bite-size-to-travel ratio (P < .0001, d = 2.6), stitch orientation (P = .014,d = 1.4), and symmetry across the incision ratio (P = .022, d = 1.3). The authors found that a simple computer algorithm can extract variables from digital images of a running suture and rapidly provide quantitative summative assessment feedback. The significant differences found between groups confirm that this system can discriminate between skill levels. This image analysis program represents a viable training tool for objectively assessing trainees' suturing, a foundational skill for many medical specialties.

The Listening Styles Profile-Revised (LSP-R): A Scale Revision and Evidence for Validity

The Listening Styles Profile (LSP-16) is the most widely used self-report listening instrument in the communication discipline. Unfortunately, researchers have utilized the instrument despite its uncharacteristically low reliability estimates and unvalidated factor structure. The following manuscript presents results from two studies designed to address these limitations. Study 1 proposes a revised measure (Listening Styles Profile-Revised; LSP-R) based on four factors: relational, analytical, task-oriented, and critical listening. Study 2 was designed to further refine and provide validity evidence for the revised scale. Internal consistency estimates and latent variable test–retest correlations showed the LSP-R to be consistent over repeated administrations; the factors were related to number of listening, information processing, empathy, communication trait, and personality variables. Beyond identifying orientations toward attending to others, the instrument developed here possesses heuristic potential for investigating the role and positive potential of listening within a variety of specific research agendas and theoretical perspectives.

Validity of scores from communication skills instruments for patients and their dental student‐clinicians

The development of appropriate communication skills by healthcare providers is central to providing quality patient-centred care. Patients can provide valuable feedback to practitioners about their clinical communication. However, in oral health care, their involvement is uncommon and instruments specific for communication in oral health care have not been available. Recently, two complementary instruments have been developed by the Faculty of Dentistry, University of Manitoba for evaluating student-clinicians' clinical communication: one for patient evaluation and one for student self-evaluation. The aim of the current study was to provide validity evidence for the scores related to the internal structure of the revised 2007 versions of these instruments in two dental clinical/education contexts, namely the Universities of Manitoba, Canada (UM) and Adelaide, Australia (UA). The proposed factor structure and loadings, and their stability across contexts were assessed using confirmatory factor analysis, and the adequacy of the internal consistency reliability of the scores was analysed using Cronbach's alpha. The factor structure of the current 2007 versions of the patient and student instruments, derived from the previously developed longer versions of these instruments, was confirmed and was consistent across the two clinical/educational contexts. A model of partial invariance provided the best fit for these data due to variations in the magnitude of the factor loadings between sites. The internal consistency reliability of scores was high with a range of 0.88-0.97. In conclusion, the current study provides preliminary evidence regarding the validity of the scores of the current 2007 instruments, in terms of the internal structure, as measuring the five factors well. Replication of the factor structure of these instrument scores with more participants at both UA and other institutions is required.

How Cognitive Interviewing can Provide Validity Evidence of the Response Processes to Scale Items

The current theory about validity reflected in the Standards for Educational and Psychological Testing (AERA et al. in Standards for educational and psychological testing, American Psychological Association, Washington, DC, 1999), offers no clear indications about the methods for gathering validity evidence about the response processes. Cognitive interviewing (CI) can play an important role answering the current demand about empirical and theoretical analyses of the response processes as a source of validity evidence in psychological testing. CI can provide validity evidence for investigating substantive aspects of construct validity and for contributing to the explanations for item and test scores (Zumbo in Handbook of statistics, vol 26, Elsevier, Amsterdam, pp. 45–79, 2007; The concept of validity: revisions, new directions and applications, IAP—Information Age Publishing Inc., Charlotte, NC, pp. 65–82, 2009). The aim of the study was to illustrate the use of cognitive interviewing method for gathering validity evidence on response processes. The search for evidence about the “response process” was guided by an argument-based approach to validity (Kane in Psychological Bulletin 1992; Educational measurement, American Council on Education/Praeger, Washington, DC, pp. 17–64, 2006). 21 cognitive interviews were carried out during the cognitive testing of the APGAR psychological scale intended to measure the “family support” construct. Cognitive interviewing provided validity evidence that explains how respondents interpret and respond to the APGAR items. Respondents maintained a shared interpretation of “family concept” while answering the APGAR scale items. Nevertheless, they included in the concept of family not only family members they live with but also other family members and even friends. CI participants were also capable of classifying their answers about the family support perception following a polythomous response system. Lastly, the role of CI in the Kane’s argument-based approach and Zumbo’s contextualized view of validity will be discussed.

Professional Practice Analysis: Determining Job Relatedness of the Certification Examination for Nurse Anesthetists

This article presents the results of the 2011 professional practice analysis (PPA) for the field of nurse anesthesia conducted by the National Board of Certification and Recertification for Nurse Anesthetists. The PPA used survey and rating scale methodologies to collect data regarding the relative emphasis on various aspects of the nurse anesthesia knowledge domain and competencies. Overall, 9,000 survey responses were analyzed by a panel consisting of individuals possessing expertise in clinical anesthesia and testing methodology, using conventional statistics and the Rasch rating scale model. Descriptions of how the survey results were used to develop test specifications are also provided. The results of this analysis provide validity evidence for the content outline and test specifications (content percentages) and thus serve as a basis of content validation for the National Certification Examination.

A Prospective, Randomized, Controlled Study Demonstrating a Novel, Effective Model of Transfer of Care between Physicians: The 5 Cs of Consultation

The objective was to evaluate whether a standardized consultation model in the emergency department (ED), the 5 Cs of Consultation (Contact, Communicate, Core Question, Collaboration, and Closing the Loop), would improve physicians' ability to relay appropriate information and communicate successfully during a consultation. This was a prospective, randomized study at a large, academic, urban, tertiary care medical center in Chicago. Forty-three emergency medicine (EM) and EM/internal medicine (EM/IM) residents were randomized into two groups, an intervention group and an unstructured group, stratified by postgraduate year (PGY). Intervention group participants received an interactive educational session on the 5 Cs of Consultation, a standardized consultation model. Intervention and unstructured groups placed two simulated consultation phone calls, based on pretested simulated patient cases, to a standardized consultant. Three raters, naive to the consultation model and blinded to group assignments, individually assessed recordings of each call using a seven-item, five-point global rating scale (GRS). Finally, an attending surgeon and an attending psychiatrist each rated respective cases using a single global rating to provide validity evidence for the scale. Residents trained with the 5 Cs model communicated significantly better, regardless of PGY and clinical case. The intervention group had significantly higher mean GRS scores than the unstructured group (4.1 vs. 3.5, F(1,39) = 33.5, p < 0.0001). Secondary analysis of the recordings suggested that encounters with more 5 Cs behaviors tended to receive higher GRS scores. A standardized educational model increased the effectiveness of consultation communication from the ED. Residents trained with the 5 Cs of Consultation scored better on consultation assessments compared with untrained residents. Training programs should consider adopting standardized consultation models.

The Nursing Student Self-Efficacy Scale

For nurse educators to optimize instruction that supports students' accurate estimates of their ability, a sound measure of nursing self-efficacy is needed. To provide evidence for an adequate item response theory model fit to scores measuring nursing students' self-efficacy to care for critically ill patients and to use the theory to provide evidence for reliable and valid interpretation of self-efficacy scores. The Nursing Student Self-Efficacy Scale was administered to 421 nursing students. Data regarding students' self-efficacy beliefs for psychomotor skill performance and communication skills were analyzed using the generalized partial credit item response theory model. All items showed acceptable model-data fit using the chi-square test with an alpha of .001. Review of item information functions showed that 40% of the items provided high information about self-efficacy and 20% provided moderately high information. Comparison of trait level distributions with reliability information given by the two subscales showed that the psychomotor subscale measured self-efficacy most accurately for second and third semester students and the communication subscale measured efficacy most accurately for first and second semester students. Significant mean differences in self-efficacy scores from both subscales were found between students in successive semesters of the program. Data analysis provided some evidence for reliable interpretation of scores indicating self-efficacy beliefs of nursing students. Comparison of scores from students in different semesters provided validity evidence; the instrument could be used to discriminate between beginning students and more advanced students possessing different levels of self-efficacy. When used for scale development, item response theory procedures can be more informative about item and test reliability than classical true score theory.

Assessment of Romantic Perfectionism

The objective of the present study was to provide validity evidence for the scores from the Romantic Relationship Perfectionism Scale. Results indicate a two-factor structure, adequate reliability, and overall good convergent, concurrent, discriminant, and incremental validity evidence. The strengths and limitations of this measure are discussed.

Manchester Clinical Placement Index (MCPI). Conditions for medical students’ learning in hospital and community placements

The drive to quality-manage medical education has created a need for valid measurement instruments. Validity evidence includes the theoretical and contextual origin of items, choice of response processes, internal structure, and interrelationship of a measure’s variables. This research set out to explore the validity and potential utility of an 11-item measurement instrument, whose theoretical and empirical origins were in an Experience Based Learning model of how medical students learn in communities of practice (COPs), and whose contextual origins were in a community-oriented, horizontally integrated, undergraduate medical programme. The objectives were to examine the psychometric properties of the scale in both hospital and community COPs and provide validity evidence to support using it to measure the quality of placements. The instrument was administered twice to students learning in both hospital and community placements and analysed using exploratory factor analysis and a generalizability analysis. 754 of a possible 902 questionnaires were returned (84% response rate), representing 168 placements. Eight items loaded onto two factors, which accounted for 78% of variance in the hospital data and 82% of variance in the community data. One factor was the placement learning environment, whose five constituent items were how learners were received at the start of the placement, people’s supportiveness, and the quality of organisation, leadership, and facilities. The other factor represented the quality of training—instruction in skills, observing students performing skills, and providing students with feedback. Alpha coefficients ranged between 0.89 and 0.93 and there were no redundant or ambiguous items. Generalisability analysis showed that between 7 and 11 raters would be needed to achieve acceptable reliability. There is validity evidence to support using the simple 8-item, mixed methods Manchester Clinical Placement Index to measure key conditions for undergraduate medical students’ experience based learning: the quality of the learning environment and the training provided within it. Its conceptual orientation is towards Communities of Practice, which is a dominant contemporary theory in undergraduate medical education.