Content-related Validity Research Articles

PCR19 Content-Related Validity of the Rand-36 Composite Scores with Other Patient-Reported Outcome Scales Among National Health and Wellness Survey Respondents in the United States, Japan, and 5 European Countries

PCR19 Content-Related Validity of the Rand-36 Composite Scores with Other Patient-Reported Outcome Scales Among National Health and Wellness Survey Respondents in the United States, Japan, and 5 European Countries

Using Automatic Item Generation to Create Multiple-Choice Questions for Pharmacy Assessment

ObjectiveAutomatic item generation (AIG) is a new area of assessment research where a set of multiple-choice questions (MCQs) are created using models and computer technology. Although successfully demonstrated in medicine and dentistry, AIG has not been implemented in pharmacy. The objective was to implement AIG to create a set of MCQs appropriate for inclusion in a summative, high-stakes, pharmacy examination. MethodsA 3-step process, well evidenced in AIG research, was employed to create the pharmacy MCQs. The first step was developing a cognitive model based on content within the examination blueprint. Second, an item model was developed based on the cognitive model. A process of systematic distractor generation was also incorporated to optimize distractor plausibility. Third, we used computer technology to assemble a set of test items based on the cognitive and item models. A sample of generated items was assessed for quality against Gierl and Lai’s 8 guidelines of item quality. ResultsMore than 15,000 MCQs were generated to measure knowledge and skill of patient assessment and treatment of nausea and/or vomiting within the scope of clinical pharmacy. A sample of generated items satisfies the requirements of content-related validity and quality after substantive review. ConclusionThis research demonstrates the AIG process is a viable strategy for creating a test item bank to provide MCQs appropriate for inclusion in a pharmacy licensing examination.

Introducing a New Model for the Analysis of Iot Business Model Ecosystem in the Field Of Rehabilitation in Iran Using the Sem Approach

Background and Aims The purpose of this article was to present a local model for ecosystem analysis of the IoT business model in the field of rehabilitation in Iran with an emphasis on improving the quality of life of the disabled community in the country. Methods The current article can be categorized as applied, combined (qualitative and quantitative), and survey-based descriptive. The society, i.e., the statistical sample, included eight experts in the qualitative part and consisted of 100 executives and experts in the quantitative part. A questionnaire was used to analyze the dimensions and indicators. The face validation of the questionnaire was performed by the Delphi technique. Content validity ratio and content validity ratio were used as two content-related validity indicators for qualitative validation. The reliability of all dimensions and indicators was confirmed through Cronbach’s test with a population consisting of 30 experts. To analyze the proposed model of the researcher, we used inferential tests, loading factor analysis, t, structure equation modeling, and fitting techniques on a 100-person sample. Results The fitness of the model was approved by experts with 99% confidence according to the philosophy, goals of the theoretical foundations of dimensions and components, feedback, and output of the model. Conclusion The findings of this paper led to a model with several dimensions: (1) environment, (2) actors, (3) ecosystem cycle, and (4) IoT architecture in business. The corresponding indicators for the identified dimensions were validated. Independence, interaction, and empowerment were considered three principle values for the target community.

The Construction of Religious Radicalism Scale (ReadS)

Religious radicalism has become an issue in Indonesia due to its link to terrorism acts. The government raised the need to conduct an assessment on radicalism. This study aims to develop the Religious Radicalism Scale (ReadS). The study was arranged into four general phases: 1) Defining religious radicalism, 2) Identifying indicators of religious radicalism, 3) Developing scale, and 4) Identifying psychometric properties. The study applied quantitative method of analysis to examine the instrument’s reliability (alpha Cronbach and omega McDonnald) and content-related validity. It involved 31 multidisciplinary scholars and professionals in identifying indicators of religious radicalism, followed by 64 participants from two contrasted groups to define psychometric properties. The findings demonstrate the efficacy of ReadS’ items in detecting religious radicalism. Moreover, ReadS achieves satisfactory reliability α=0.97 and ω=0.97; hence, the instrument can be applied in investigating religious radicalism. The study may help academics and practitioners in scientifically examining religious radicalism.

Assessing People’s Willingness to Participate in Counter Violent Extremism Programmes: The development of Counterradicalism Readiness Scale (CoRS)

Counterterrorism approach in Indonesia emphasizes on prevention rather than prosecution. Radicalism could lead to the acts of terrorism; hence, in terms of terrorism prevention, the role of civil society is significant in avoiding radicalization of people. Here lies the urgency of community empowerment. As an instrument to examine people’s readiness in preventing terrorism is unclear, this study was held. The study intends to formulate an instrument for measuring counterradicalism readiness (called ‘CoRS’). It aimed to evaluate people’s readiness to be involved in counterradicalism and CVE (Counter Violent Extremism) programs. The development of CoRS was organized into detailed steps: 1) defining counterradicalism readiness and its aspects, 2) identifying behaviour indicators of counterradicalism readiness, 3) making items for scale, 4) performing items’ readability test, 5) conducting pilot study, and 6) holding statistical analysis. Quantitative method was applied in this study by involving 19 security practitioners to assess CoRS’ content-related validity and 227 participants for the instrument’s reliability (with Alpha Cronbach and Omega McDonnald). The findings demonstrate the efficacy of CoRS’ items in measuring counterradicalism readiness. Furtmermore, CoRS reaches satisfactory reliability α=0.92 and ω=0.93. The study may assist practitioners in identifying people’s readiness and willingness to participate in counterradicalism programs.

Development and content validation of a questionnaire for functional movement disorders

Von Kriegszitterern zu Teenies mit TikTok-Tics – funktionelle Bewegungsstörungen galten lange als motorischer Ausdruck seelischer Turbulenzen. Tatsächlich werden psychodynamische Modelle jedoch zunehmend durch neurophysiologische Erkenntnisse ergänzt, sodass die Kombination von Psychotherapie mit spezialisierter Physiotherapie an Bedeutung gewinnt. Bislang existiert allerdings kein störungsspezifisches Ergebnismaß, welches patientenrelevante Aspekte dieser vielfältigen Störung adäquat abbildet. Ein erster derartiger Fragebogen wurde entwickelt und in einem mehrstufigen Entwicklungsprozess inhaltlich validiert. Relevanz und Verständlichkeit der Items wurden zuerst durch ein Expertengremium und anschließend durch betroffene Patienten evaluiert und entsprechend angepasst. Der so konstruierte Fragebogen erlaubt erstmals eine inhaltlich valide Quantifizierung der subjektiven Beschwerden und Auswirkungen funktioneller Bewegungsstörungen. Als Nächstes erfolgt eine multizentrische teststatistische Validierung.Zusatzmaterial onlineDie Online-Version dieses Beitrags (10.1007/s00115-021-01247-1) enthält weitere Infomaterialien. Beitrag und Zusatzmaterial stehen Ihnen auf www.springermedizin.de zur Verfügung. Bitte geben Sie dort den Beitragstitel in die Suche ein, das Zusatzmaterial finden Sie beim Beitrag unter „Ergänzende Inhalte“.

Developing Innovative Virtual Reality Simulations to Increase Health Care Providers' Understanding of Social Determinants of Health.

Health care providers (HCPs) who work primarily with Medicaid patients must be competent in identifying and addressing social determinants of health (SDH). A curricular gap exists between promoting an understanding of SDH and teaching HCPs how to recognize and increase empathy to manage them. The project aim was to develop two virtual reality simulations (VRSs) as innovative methods to teach HCPs to identify and manage SDH. A secondary aim was to decrease unconscious bias and increase empathy by experiencing SDH from their patients' perspective. Scripts for two VRSs were created by two HCP educators and clinicians. Scripts were evaluated by experts using an index of content validity (CVI). An advisory panel critiqued the scripts for appropriateness for VRSs, adequacy of evidence-based practice, and use of VRS equipment and software. The panel participated in a focus group and completed a final evaluation. The VRSs then were pilot tested with five HCPs who assessed content and utility and participated in interviews. This led to iterative improvements. Qualitative data were analyzed using a content analysis approach. The VRS scripts demonstrated adequate content-related validity evidence with CVI scores of 0.81 and 0.75. The expert panel found the VRS easy to use, useful as an educational tool, while promoting empathy for patients. Overall, participants were satisfied with using the VRS as an educational experience. Through VRS technology, this project addresses a curricular gap in HCP training on SDH. VRS can be a useful tool to increase HCPs' understanding of SDH and, potentially, their empathy for patients.

Supporting Self-Directed Learning: Development of a Faculty Evaluation Scale

Construct: We sought to assess medical education faculty members’ ability to support students in the development of self-regulated learning skills. Background: It is broadly agreed that medical students should become self-directed lifelong learners to succeed as physicians. To that end, many instruments have been developed that measure student attributes of self-directed learning (SDL). By contrast, no such analogous valid instrument is available to assess faculty familiarity in pedagogical strategies to promote SDL among students. Approach: An item bank with 45 items was created and its content-related validity evaluated by a panel of twenty experts. The items were selected and validated in the framework of Item Response Theory (IRT). The unidimensionality of all items within four constructs was assessed by using modified parallel analysis and 2 parameter IRT model calibration. The final version of the Recognition of Learning-Oriented Teaching Strategies (RoLOTS) instrument was validated by using the IRT marginal reliability as well as 2PL model calibrations. Different Item Functioning (DIF) in student-educators and gender was examined. Findings: The final version of RoLOTS included 20 items, with five in each of four subdomains: building content knowledge; the emotional and motivational aspects of learning; leveraging the social nature of the learning process; and metacognitive processes that promote student regulation of the learning process. Construct validity and reliability of each of the four domains were well supported by the results. Significant DIF was not detected in student-educator and gender. Conclusions: The RoLOTS successfully evaluated whether a faculty member is familiar with pedagogical tools to promote medical students’ self-directed learning, which can be used as a first step in needs-based professional development. Further research is needed to provide more validity and reliability evidence among groups with diverse SDL experience.

Challenges in Assessment Centres: Lessons from Experience

The success of the assessment centre (AC) approach has been widely acknowledged and its widespread practice testifies to this. In India, its adoption as a tool for HRD is increasing slowly and steadily. Its proven success is due to its strong criterion and content-related validity. However, certain challenges during design and development of AC can undermine its very strengths. Using an interpretative, qualitative approach, this article builds on the first-hand experiences of the authors as well as 26 other Indian assessors in conducting ACs to highlight the issues faced at 3 different levels of the AC methodology—design, implementation, and inferential. It also highlights the interconnectedness between these issues at various levels and suggests interventions to minimize assessment errors. The article adds on to the AC literature by detailing the challenges faced in ACs in a developing country, which had rarely been done before. This study holds significant relevance for assessors, consulting agencies providing assessment services, and even to client organizations envisaging AC for their employees.

Development of the Quality of Life in Youth Services Scale (QOLYSS): Content-Related Validity Evidence Based on Adolescents’ and Expert Reviewers’ Perspectives

Despite increased interest in the quality of life of younger populations, the quality of life of youngsters in vulnerable situations such as those in youth services remains under-researched. To gain insight into critical ingredients for understanding and enhancing the lives of these youngsters, appropriate assessment tools are needed. This study focuses on providing content-related validity evidence for the new Quality of Life in Youth Services Scale (QOLYSS) for adolescents in youth services. A sequential approach to item generation, refinement, and evaluation is presented. The initial item bank is devised by means of a thorough literature search and evaluated against findings from qualitative research among adolescents in youth services. In addition, a three-round international expert panel (N = 5) is convened to align identified items to the adopted quality of life framework, and to provide evidence on domain representation and relevance. Last, input is collected from adolescents in youth services themselves. More specifically, 20 group discussions and 13 individual interview sessions are organized to support 21 adolescents in critically assessing the content of the item bank on level of relevance, clarity, and coverage. The expert panel resulted in a provisional scale of 262 unique items that, based on adolescents’ input, is further reduced to a set of 88 items. Adolescents omitted items because of divergent reasons (e.g., content-overlap, self-evident character of the items, level of irrelevance). Explanatory qualitative information on the item selection process is provided. This study provides a detailed understanding of the content-related validity evidence of a new multidimensional quality of life scale. Moreover, it highlights the importance of acknowledging youngsters as critical partner in quality of life research and assessment.

Cultural Intelligence and Organizational Capabilities: An Appraisal of Jordanian Commercial Banks

The purpose of this research is to identify relationship between cultural intelligence and organizational capabilities in Jordanian Commercial Banks. The research method is descriptive- correlation. Statistical population included the supervisory and regulatory functions in these banks which totaling (524) employees, and the sample size of (368) employees was determined based on the Cochran formula. The sampling method used is proportional stratified random sampling. Data collected were used from two questionnaireswith these items: cultural intelligence questionnaire of Ahn & Ettner and organizational capabilities questionnaire of Al-Ajami. For appointment of questionnaires validity, of the method content-related validity, and to assess scale the reliability of the questionnaires, Cronbach's Alpha coefficient was used. It was estimated 0.94 and 0.78 respectively. Results situation of cultural intelligence and organizational capabilities was above the average level. The findings indicated that there is a significant positive relationship between cultural intelligence and its dimensions with organizational capabilities, there is a significant positive relationship between cultural intelligence and its dimensions with marketing capabilities, there is a significant positive relationship between cultural intelligence and its dimensions with technological capabilities, and there is a significant positive relationship between cultural intelligence and its dimensions with managerial capabilities in Jordanian commercial banks. Under the light of the results achieved, the study presented a number of recommendations, the most important recommendations include:Individuals in the Jordanian commercial banks should to have awareness of cultural diversity, a high level of understanding of the differences between the diverse cultures and the ability to analyze cultural elements in the work environment, which leading to enhance what possesses these banks of resources, capabilities and competencies that needs to improve its performance and to achieve its objectives and its mission.

Use of a Low-Cost Portable 3D Virtual Reality Gesture-Mediated Simulator for Training and Learning Basic Psychomotor Skills in Minimally Invasive Surgery: Development and Content Validity Study.

BackgroundSimulation in virtual environments has become a new paradigm for surgeon training in minimally invasive surgery (MIS). However, this technology is expensive and difficult to access.ObjectiveThis study aims first to describe the development of a new gesture-based simulator for learning skills in MIS and, second, to establish its fidelity to the criterion and sources of content-related validity evidence.MethodsFor the development of the gesture-mediated simulator for MIS using virtual reality (SIMISGEST-VR), a design-based research (DBR) paradigm was adopted. For the second objective, 30 participants completed a questionnaire, with responses scored on a 5-point Likert scale. A literature review on the validity of the MIS training-VR (MIST-VR) was conducted. The study of fidelity to the criterion was rated using a 10-item questionnaire, while the sources of content-related validity evidence were assessed using 10 questions about the simulator training capacity and 6 questions about MIS tasks, and an iterative process of instrument pilot testing was performed.ResultsA good enough prototype of a gesture-based simulator was developed with metrics and feedback for learning psychomotor skills in MIS. As per the survey conducted to assess the fidelity to the criterion, all 30 participants felt that most aspects of the simulator were adequately realistic and that it could be used as a tool for teaching basic psychomotor skills in laparoscopic surgery (Likert score: 4.07-4.73). The sources of content-related validity evidence showed that this study’s simulator is a reliable training tool and that the exercises enable learning of the basic psychomotor skills required in MIS (Likert score: 4.28-4.67).ConclusionsThe development of gesture-based 3D virtual environments for training and learning basic psychomotor skills in MIS opens up a new approach to low-cost, portable simulation that allows ubiquitous learning and preoperative warm-up. Fidelity to the criterion was duly evaluated, which allowed a good enough prototype to be achieved. Content-related validity evidence for SIMISGEST-VR was also obtained.

Initial Validation of a Patient-Reported Measure of Compassion: Determining the Content Validity and Clinical Sensibility among Patients Living with a Life-Limiting and Incurable Illness.

Although compassionate care is considered a cornerstone of quality palliative care, there is a paucity of valid and reliable measures to study, assess, and evaluate how patients experience compassion/compassionate care in their care. The aim was to develop a patient-reported compassion measure for use in research and clinical practice with established content-related validity evidence for the items, question stems, and response scale. Content validation for an initial 109 items was conducted through a two-round modified Delphi technique, followed by cognitive interviews with patients. A panel of international Subject Matter Experts (SMEs) and a Patient Advisory Group (PAG) assessed the items for their relevancy to their associated domain of compassion, yielding an Item-level Content Validity Index (I-CVI), which was used to determine content modifications. The SMEs and the PAG also provided narrative feedback on the clarity, flow, and wording of the instructions, questions, and response scale, with items being modified accordingly. Cognitive interviews were conducted with 16 patients to further assess the clarity, comprehensibility, and readability of each item within the revised item pool. The first round of the Delphi review produced an overall CVI of 72% among SMEs and 80% among the PAG for the 109 items. Delphi panelists then reviewed a revised measure containing 84 items, generating an overall CVI of 84% for SMEs and 86% for the PAG. Sixty-eight items underwent further testing via cognitive interviews with patients, resulting in an additional 14 items being removed. Having established this initial validity evidence, further testing to assess internal consistency, test-retest reliability, factor structure, and relationships to other variables is required to produce the first valid, reliable, and clinically informed patient-reported measure of compassion.

Development and Validation of a Vertical Scale for Formative Assessment in Mathematics

The regular formative assessment of students' abilities across multiple school grades requires a reliable and valid vertical scale. A vertical scale is a precondition not only for comparing assessment results and measuring progress over time, but also for identifying the most informative items for each individual student within a large item bank independent of the student's grade to increase measurement efficiency. However, the practical implementation of a vertical scale is psychometrically challenging. Several extant studies point to the complex interactions between the practical context in which the scale is used and the scaling decisions that researchers need to make during the development of a vertical scale. As a consequence, clear general recommendations are missing for most scaling decisions. In this study, we described the development of a vertical scale for the formative assessment of third- through ninth-grade students' mathematics abilities based on item response theory methods. We evaluated the content-related validity of this new vertical scale by contrasting the calibration procedure's empirical outcomes (i.e., the item difficulty estimates) with the theoretical, content-related item difficulties reflected by the underlying competence levels of the curriculum, which served as a content framework for developing the scale. Besides analyzing the general match between empirical and content-related item difficulty, we also explored, by means of correlation and multiple regression analyses, whether the match differed for items related to different curriculum cycles (i.e., primary vs. secondary school), domains, or competencies within mathematics. The results showed strong correlations between the empirical and content-related item difficulties, which emphasized the scale's content-related validity. Further analysis showed a higher correlation between empirical and content-related item difficulty at the primary compared with the secondary school level. Across the different curriculum domains and most of the curriculum competencies, we found comparable correlations, implying that the scale is a good indicator of the math ability stated in the curriculum.

Development of the University of Wisconsin Running Injury and Recovery Index.

Runners experience a high proportion of overuse injuries, with extended recovery periods involving a gradual, progressive return to preinjury status. A running-specific patient-reported outcome (PRO) measure does not exist, and a questionnaire assessing critical elements of runners' recovery processes may have excellent psychometric properties. To develop a valid, reliable, and responsive evaluative PRO measure to assess longitudinal change in running ability after running-related injury (RRI) for clinical practice and research applications. Self-identified runners and selected experts participated in an iterative, 6-step development process of the University of Wisconsin Running Injury and Recovery Index (UWRI) in this longitudinal clinical measurement study. Content-related validity was assessed using open comments. Reproducibility was assessed using Cronbach's alpha, the intraclass correlation coefficient (ICC), and standard error of measurement (SEM). An anchor-based construct validity assessment measured the association between the change in UWRI score and global rating of change (GROC). Responsiveness assessments included floor and ceiling effects. The 9-item UWRI assesses running ability following an RRI, with the maximum score of 36 indicating a return to preinjury running ability. The UWRI demonstrated acceptable internal consistency (α = .82), test-retest reliability (ICC = 0.93), and SEM (1.47 points). Change in UWRI score was moderately correlated with the GROC (r = 0.61; 95% confidence interval: 0.4, 0.76). Floor and ceiling effects were absent. Completion required 3 minutes 15 seconds. The UWRI is a reliable PRO measure and is responsive to changes in running function following an RRI, with minimal administrative burden. Therapy, level 2c. J Orthop Sports Phys Ther 2019;49(10):751-760. Epub 3 Aug 2019. doi:10.2519/jospt.2019.8868.

Development and Validation of a Women's Financial Self-Efficacy Scale

Lack of standardized measurement is one of the main factors that inhibits rigorous evaluations of financial literacy programs. However, although several scholars have developed financial self-efficacy measurements, none have been tailored for women. This article aims to develop and validate a Women's Financial Self-Efficacy Scale (WFSES). Results showed that the WFSES had an excellent reliability coefficient alpha (.93). The scale had good content-related validity, which covered all key domains in financial management for women. The criterion-related validity showed that the WFSES was positively correlated with the New General Self-Efficacy Scale (NGSES). Factor analysis showed four factors to be consistent with the common categories in financial management curricula.

Development and Validation of the Career Development Sentence Completion Test for Elementary School Career Counseling

The purpose of this study is to develop and verify the validity of the Career Development Sentence Completion Test (CDSCT) for elementary school students. Elementary school homeroom teachers or school counselors can use the CDSCT when they want to understand students’ career development and identify individual career concerns. The process of the CDSCT development was completed in four phases. Phases one and two involved formulating the CDSCT design, writing an initial pool of 80 items from questions that the elementary school teachers reported using in career counseling, and reviewing the items for revision and deletion. These steps provided evidence of content-related validity by experts. For the third phase, apilot test was conducted with 10 5th grade students in elementary school to establish face validity. Finally, 23items were selected for CDSCT for elementary school career counseling. In the fourth phase, 57 6th grade students in elementary schools were administered thenewly developed CDSCT and the Career Awareness Scale for elementary school students with the interval of one week between the two tests. The relationship between all four domains of the CDSCT and the Career Awareness Scale was significant verifying that the CDSCT for the elementary school students has convergent validity. The CDSCT is a useful instrument for understanding children’s career development process and as a tool for individual career counseling.

Development and Testing of an Addiction Treatment Level of Care Determination Tool.

Two studies examined inter-rater reliability and content-related validity of an addiction treatment level of care determination tool currently in use in New York, the LOCADTR 3.0. The studies occurred after tool implementation. In study 1, 139 providers used the LOCADTR 3.0 to determine level of care for four case vignettes. Inter-rater reliability coefficients were calculated. In study 2, 387,338 state records from existing data were analyzed to determine how often providers opted to override the LOCADTR 3.0 level of care determination by choosing an alternative level of care. In study 1, an acceptable inter-rater reliability (IRR = .57-.59) was found. Good indication of content-related validity was also found; participants chose the same level of care the study team chose for each vignette 80% of the time. In study 2, the override option was selected only 10% of the time, further establishing the content validity of the tool. These studies provide evidence for acceptable preliminary reliability and validity of the LOCADTR 3.0.

Establishing Content-Related Validity Evidence for Assessments in Counseling: Application of a Sequential Mixed-Method Approach

Establishing content-related validity evidence is considered essential to draw conclusions about a scale’s quality. The purpose of this conceptual article is to provide best practice recommendations in the content validation procedure of scale development in the field of counseling, incorporating both qualitative and quantitative content review phases as a sequential mixed-method approach. In line with this goal, a description of the four basic sequential stages of the content validation procedure is provided, including a brief overview of content validity and Lawshe’s (1975) technique as a quantitative approach to content validity.

Assessing Social Cognition of Persons with Schizophrenia in a Chinese Population: A Pilot Study.

Social cognition is a core limiting factor of functional recovery among persons with schizophrenia. However, there is a lack of standardized and culturally relevant assessment tools for evaluating social cognitive performance in Chinese persons with schizophrenia. The purposes of this study were to (1) develop and validate two social cognitive instruments, the Chinese Facial Emotion Identification Test (C-FEIT) and the Chinese Social Cognition and Screening Questionnaire (C-SCSQ), that assess three key domains of social cognition and (2) to evaluate preliminary psychometric properties of the two assessments. The results demonstrated that the C-FEIT and the social cognitive subscales of C-SCSQ possess satisfactory content-related validity and test–retest reliability (ICC ranging from 0.76 to 0.85). Subscales of the C-FEIT and the C-SCSQ showed low to medium correlation with two concurrent neurocognitive measures (absolute values of r ranging from 0.22 to 0.45) and concurrent measures of functional performance (absolute values of r ranging from 0.22 to 0.46). Our findings generally support the use of the C-FEIT and the C-SCSQ as reliable and valid tools for assessing emotion perception, theory of mind (intention-inferencing), and hostile attributional style, which are the key outcome indicators of social cognitive interventions for persons with schizophrenia.