Classical Test Theory Analysis Research Articles

The development of the reading comprehension—Progress monitoring (RC-PM) tool for late elementary students

Notwithstanding reading comprehension is a key competence in today’s society, many late elementary students struggle with it. In this respect, effective instructional incentives are required to foster students’ reading comprehension. However, appropriate assessment instruments to monitor students’ reading comprehension on a regular basis and to make substantiated instructional decisions are lacking. Therefore, a Reading Comprehension – Progress Monitoring tool was developed, consisting of six parallel tests equivalent in difficulty and length. To this aim, classical test theory analyses, item response theory analyses, and automated test assembly were conducted (n = 3,269 students). Suggestions for future research and practice are discussed.

Development and Psychometric Properties of Self-Reported Job Interview Skills and Job Interview Anxiety for Autistic Transition-Age Youth.

The study of job interview training is an emerging area among transition-age autistic youth who face significant challenges when navigating job interviews. The autism field has limited measures that have undergone rigorous psychometric evaluation. We sought to evaluate the psychometric properties of adapted self-report measures assessing job interview skills and job interview anxiety. As part of two parent randomized controlled trials, eighty-five transition-age autistic youth completed measures related to the strength of their job interview skills and their level of job interview anxiety. We conducted classical test theory analyses, exploratory and confirmatory factor analyses, and Rasch model analytic and calibration analyses. Pearson correlations were used to establish concurrent, divergent, and criterion validity by correlating these scales with measures of social differences, depressive symptoms, behaviors, neuropsychological functioning, and work history. Our analyses yielded two brief and reliable scales: Measure of Job Interview Skills (MOJO-iSkills) and Measure of Job Interview Anxiety (MOJO-iAnxiety), which demonstrated initial concurrent, divergent, and criterion validities when correlated with measures of depressive symptoms, social differences, internalizing and externalizing behavior, and work history. This study presents initial evidence that MOJO-iSkills and MOJO-iAnxiety have acceptable psychometric properties supporting they can be used to reliably and validly assess job interview skills and interview anxiety.

Diseño y validación de una escala de adherencia a antineoplásicos orales

ObjectiveTo design and validate a questionnaire to measure adherence to oral antineoplastic drugs. The availability of a simple, validated tool that can be applied to routine care will make it possible to detect and identify non-adherence in order to establish strategies to improve adherence and optimize the quality of healthcare services. MethodValidation study of the questionnaire designed to assess adherence to antineoplastic drugs in a sample of outpatients who collect their medication in two Spanish hospitals. Its validity and reliability will be analyzed, based on a previous qualitative methodology study, using classical test theory and Rasch analysis. We will examine its performance, item fit, response structure and person fit to the predictions of the model, as well as dimensionality, item-person reliability, the appropriateness of the level of difficulty of the items to the sample, and the differential performance of the items according to gender.

Translated article] Design and validation of a scale to measure adherence to oral antineoplastic drugs

ObjectiveDesign and validate a scale to measure adherence to oral antineoplastic drugs. The availability of a simple, validated tool that can be applied to routine care will make it possible to detect and identify non-adherence in order to establish strategies to improve adherence and optimize the quality of healthcare services. MethodValidation study of the scale designed to assess adherence to antineoplastic drugs in a sample of outpatients who collect their medication in two Spanish hospitals. Its validity and reliability will be analyzed, based on a previous qualitative methodology study, using classical test theory and Rasch analysis. We will examine its performance, item fit, response structure and person fit to the predictions of the model, as well as dimensionality, item-person reliability, the appropriateness of the level of difficulty of the items to the sample, and the differential performance of the items according to gender.

Psychometric Evaluation of the Spanish Families Importance in Nursing Care: Nurses' Attitudes Scale Through Classical Test Theory and Rasch Analysis.

Nurses' attitudes toward families play an important role in improving relationships with patients' families. It is essential to have valid and reliable instruments to assess nurses' attitudes toward involving families. The aim of this study was to analyze the psychometric properties of the refined Spanish version of the Families' Importance in Nursing Care-Nurses' Attitudes (FINC-NA) according to classical test theory and the Rasch model (N = 263). Cronbach's alpha values were .73 to .87 and intraclass correlation coefficients ranged from .72 to .86. Rasch analysis results suggested that it was a multidimensional scale with four dimensions and a simpler response scheme than the original scale. Except for one item, the scale was free from bias regarding age and experience time. The FINC-NA is a reliable and valid measure showing a good fit to the Rasch model and is ready to map nurses' attitudes and measure the effectiveness of family nursing educational interventions.

Analyzing the science achievement test: Perspective of classical test theory and Rasch analysis

Assessment in teaching and learning allows both instructor and students to evaluate the achievement, performance and improve upon it. The achievement test appears as one of the most commonly used assessment tools to determine students' gains in the cognitive domain. The study analysed science achievement test (SAT), a multiple-choice test for grade 10 senior secondary school students. The 50 SAT items were developed in compliance with the Malaysian science curriculum specification. The face and content validity of SAT were validated by experts in the science field. The SAT was administered with 50 students in a pilot study. The collected data was analysed using item analysis based on classical test theory (CTT) and Rasch analysis. In terms of difficulty index (p), item analysis showed that there were five difficult items, 33 moderately difficult items, and 12 easy items. Findings also reported eight items presented as poor items due to the poor discriminating power. The SAT showed a good coefficient of 0.862 for Kuder-Richardson 20 (KR-20) and 0.851 for the split half method. While the Rasch analysis showed a reliable and good separation for both item and person. Besides, the Rasch analysis displayed that there are two items should be refreshed or removed. The study revealed that the SAT was a solid, valid, and reliable tool, which is suitable to be used to measure the student’s achievement in science.

Psychometric Characteristics of the Culture Fair Intelligence Test Scale 2

This study aims to investigate the psychometric characteristics of the CFIT-scale-2, including investigation of difficulty level, discrimination, distractor effectiveness, DIF (Differential item functioning), IRT (Item Response Theory) 2PL model, validity, and reliability. Respondents were 507 students with 245 (48%) females and 262 (52%) males, with ages ranged from 8-13 years (M= 10.88; SD = 1.35). The classical test theory (CTT) analysis showed that the item difficulty level on the CFIT-scale-2 had varying item difficulty levels, from easy to difficult. However, the test arrangement was not structured according to the suggested difficulty level from easy to medium to difficult. The discrimination of items was poor because 28 items were not included in the very good category (p > .40). In addition, 47 (25%) of the 184 distractors are ineffective. CFIT-scale-2 did not contain DIF (Adj.p > .05), and IRT analysis showed that the CFIT-scale-2 was not structured according to the difficulty pattern from easy, medium, and difficult. The CFIT-scale-2 based on IRT analysis contained 44 items or 96% with good discrimination. The results of a construct validity test using the CFA technique showed a good fit model (p < .001; RMSEA < .08; SRMR < .08), which was acceptable and supported the fit between the theoretical and empirical model. The reliability coefficient value of Cronbach's alpha was 0.88 (α > .70), indicating that the CFIT-scale-2 had good reliability, but the construct reliability was below an acceptable value (CR < .69). According to this study, the psychometric characteristics of the CFIT-scale-2 should be revised and reevaluated.

Beyond item analysis: Connecting student behaviour and performance using e‐assessment logs

AbstractTraditional item analyses such as classical test theory (CTT) use exam‐taker responses to assessment items to approximate their difficulty and discrimination. The increased adoption by educational institutions of electronic assessment platforms (EAPs) provides new avenues for assessment analytics by capturing detailed logs of an exam‐taker's journey through their exam. This paper explores how logs created by EAPs can be employed alongside exam‐taker responses and CTT to gain deeper insights into exam items. In particular, we propose an approach for deriving features from exam logs for approximating item difficulty and discrimination based on exam‐taker behaviour during an exam. Items for which difficulty and discrimination differ significantly between CTT analysis and our approach are flagged through outlier detection for independent academic review. We demonstrate our approach by analysing de‐identified exam logs and responses to assessment items of 463 medical students enrolled in a first‐year biomedical sciences course. The analysis shows that the number of times an exam‐taker visits an item before selecting a final response is a strong indicator of an item's difficulty and discrimination. Scrutiny by the course instructor of the seven items identified as outliers suggests our log‐based analysis can provide insights beyond what is captured by traditional item analyses. Practitioner notesWhat is already known about this topic Traditional item analysis is based on exam‐taker responses to the items using mathematical and statistical models from classical test theory (CTT). The difficulty and discrimination indices thus calculated can be used to determine the effectiveness of each item and consequently the reliability of the entire exam. What this paper adds Data extracted from exam logs can be used to identify exam‐taker behaviours which complement classical test theory in approximating the difficulty and discrimination of an item and identifying items that may require instructor review. Implications for practice and/or policy Identifying the behaviours of successful exam‐takers may allow us to develop effective exam‐taking strategies and personal recommendations for students. Analysing exam logs may also provide an additional tool for identifying struggling students and items in need of revision.

“Development and validation of the ICAP Technology Scale to measure how teachers integrate technology into learning activities”

Previous research investigating the use of technology in school has focused mainly on the frequency of use of digital tools during lessons rather than investigating how technology is integrated with respect to different kinds of learning activities.Since the impact of technology use on learning depends on how it is used and on what activities supported by technology are implemented in lessons, a measurement instrument assessing how technology is integrated into learning activities is necessary to investigate its impact on teaching and learning processes. According to the interactive, constructive, active, and passive (ICAP) framework, which distinguishes four different learning activities based on the level of students' cognitive engagement, we developed the 12-item ICAP Technology Scale (ICAP-TS) that accounts for all four dimensions of technology integration in lessons. We used confirmatory factor analysis to validate the four-factor structure of the ICAP-TS with a sample of 1059 upper-secondary school teachers from Switzerland. We also examined reliability using classical test theory and Rasch model analysis to assess the scale's psychometric characteristics. We then analyzed the associations between the ICAP-TS and a general use frequency measure of 12 educational technologies to test the criterion validity. The results confirmed the four-factor structure of the ICAP-TS and revealed good instrument accuracy. The most difficult items to endorse are those describing the integration of technology into interactive learning activities. Furthermore, all 12 items significantly correlated with the frequency of use of 12 educational technologies. We recommend the ICAP-TS as a short and reliable measurement scale for assessing how technology is integrated into lessons, considering different learning activities based on the ICAP theoretical model.

Pamukkale eleştirel düşünme becerisi ölçeği: geçerlik ve güvenirlik çalışması

The aim of this study is to develop a valid and reliable measurement tool that measures critical thinking skills of university students. Pamukkale Critical Thinking Skills Scale was developed as two separate forms; multiple choice and open-ended. The validity and reliability studies of the multiple-choice form were constructed on two different theoretical frameworks as classical test theory and item-response theory. According to classical test theory, exploratory and confirmatory factor analyses were performed, to item-response theory, the Generalized Partial Credit Model (GPCM) for one-dimensional and multi-category scales was tested for the construct validity of the multiple-choice form of the scale. Analysis results supported the unidimensional structure of the scale. The reliability analyzes showed that the internal consistency coefficient of the scale and the item-total correlation values were high enough. The test-retest analysis results supported that the scale shows stability over time regarding the field it measures. The results of the item-response theory-based analysis also showed that the scale met the item-model fit assumptions. In the evaluation of the open-ended form of the scale, a rubric was used. Several studies were conducted on the validity and reliability of the open-ended form, and the results of the analysis provided psychometric support for the validity and reliability. As a result, Pamukkale Critical Thinking Skills Scale, which was developed in two forms, is a valid and reliable measurement tool to measure critical thinking skills of university students. The findings were discussed in the light of the literature and some suggestions were given.

Adaptation and Validation of the Malay Version of the SAVE-9 Viral Epidemic Anxiety Scale for Healthcare Workers.

Viral epidemics have surfaced frequently over the past quarter-century, with multiple manifestations of psychological distress. This study sought to establish the psychometric properties of the Malay version of SAVE-9 among healthcare workers. A total of 203 healthcare workers across Malaysia participated in the research. The Malay version of SAVE-9 was translated and back-translated using the WHO instrument validation protocols. Classical Test Theory (CTT) and Rasch analysis were used to assess the validity and reliability of the Malay version of the SAVE-9 scale. The analysis was run using IBM SPSS 26.0 and JAPS. Cronbach’s alpha was used to measure the internal consistency of SAVE-9, which was found to be satisfactory (Cronbach’s α = 0.795). The correlations between the SAVE-9 and other measured scales (GAD-7 and PHQ-9) were statistically significant. A score of 22 was defined as a cut-off point with good sensitivity (0.578) and specificity (0.165). The Malay version of the Stress and Anxiety to Viral Epidemics-9 (SAVE-9) scale is valid and reliable after testing among healthcare workers. It is psychometrically suitable to be used in assessing healthcare workers’ stress and anxiety specific to viral epidemics.

Measuring a Patient's understanding of their prognosis: An exploratory analysis.

In patients with cancer across the illness trajectory, treatment decisions are often influenced by one's perception of their prognosis (i.e., curability of disease, life expectancy, quality of life). However, research on how patients understand their prognosis (i.e., prognostic understanding) has been limited by simplistic measurement tools that fail to capture the complexity of the construct. This study describes the initial development of a measure of prognostic understanding: the Prognostic Understanding Perceptions Scale (PUPS) for use in patients with advanced cancer. An initial pool of 16 candidate items were developed through semi-structured interviews with 15 experts (oncology, psycho-oncology and palliative care professionals) and 30 advanced cancer patients. We investigated the dimensionality, internal item structure, item difficulty and item discrimination of the item pool using exploratory factor analysis (EFA), classical test theory (CTT) and item response theory (IRT) analyses. Convergent and divergent validity were based on correlations between PUPS, terminal illness acknowledgement, and self-report measures of depression, anxiety, hopelessness, and death acceptance. The final measure was comprised of nine items encompassing three factors (perceived curability, illness trajectory, treatment options), yielding strong psychometric properties. These results provide preliminarily support for PUPS as a multifaceted measure of prognostic understanding developed for use in patients with advanced cancer. Preliminary findings also highlight the potential utility of the PUPS for clinical settings, as a means of enhancing communication between patients and physicians.

The Impact of the Appalachian Career Training in Oncology (ACTION) Program on High School Participants

Kentucky ranks first in the nation in cancer incidence and mortality. The Appalachian region of the state experiences the highest cancer disparities due to inequities in many social determinants of health as well as poor health behaviors. As a strategy for addressing cancer and education disparities in the region, the Appalachian Career Training in Oncology (ACTION) Program at the University of Kentucky Markey Cancer Center engages Appalachian-native high school students in cancer education, research, and outreach activities. Entry and exit surveys were administered to participants. Classical test theory and exploratory factor analysis were used to examine the instruments used for program evaluation, whereas repeated measures ANOVA was used to determine if there were significant differences in means between entry and exit timepoints. There was an increase in students' understanding and comfort with cancer-related topics between entry and exit survey administrations. Students indicated that the program improved their research skills and career planning skills. Furthermore, students had positive perceptions of all aspects of the program. Data suggest that ACTION has a positive impact on high school students. Specifically, results show increased understanding and comfort with cancer-related topics, research skills, and career planning.

Adaptation and Validation of the Malay Version of the Stress and Anxiety to Viral Epidemics-6 Items Scale Among the General Population.

The coronavirus disease pandemic has caused untold distress owing to both its physical and psychological sequelae, and such distress is further exacerbated by multiple socioeconomic ramifications. The Stress and Anxiety to Viral Epidemics-6 Items (SAVE-6). Scale was developed to specifically assess the anxiety response of the general population to viral epidemics. This study aimed to establish the psychometric properties of the Malay version of the SAVE-6 Scale in the general population. Herein, a total of 257 individuals participated. World Health Organization instrument validation protocols were used to translate and back-translate the Malay SAVE-6 Scale. Subsequently, the classical test theory and Rasch analysis were used to ascertain the validity and reliability of the scale. Cronbach α was used to measure the internal consistency, which was found to be satisfactory (α = 0.866). The correlations between the SAVE-6 Scale and other scales, including the Generalized Anxiety Disorder-7 Items Scale (r = 0.421, p < 0.001) and Patient Health Questionnaire-9 Items (r = 0.354, p < 0.001) were significant. Taken together, the Malay version of the SAVE-6 Scale is valid and reliable for use in the general population and is psychometrically suitable for assessing stress and anxiety specific to viral epidemics.

Using item response theory to appraise key feature examinations for clinical reasoning

Background Validation of examinations is usually based on classical test theory. In this study, we analysed a key feature examination according to item response theory and compared the results with those of a classical test theory approach. Methods Over the course of five years, 805 fourth-year undergraduate students took a key feature examination on general medicine consisting of 30 items. Analyses were run according to a classical test theory approach as well as using item response theory. Classical test theory analyses are reported as item difficulty, discriminatory power, and Cronbach’s alpha while item response theory analyses are presented as item characteristics curves, item information curves and a test information function. Results According to classical test theory findings, the examination was labelled as easy. Analyses according to item response theory more specifically indicated that the examination was most suited to identify struggling students. Furthermore, the analysis allowed for adapting the examination to specific ability ranges by removing items, as well as comparing multiple samples with varying ability ranges. Conclusions Item response theory analyses revealed results not yielded by classical test theory. Thus, both approaches should be routinely combined to increase the information yield of examination data.

Assessment of Loneliness During the Pandemic: Comparing Various Short Forms of the UCLA Loneliness Scale in South Africa Using Classical Test Theory and Mokken Analysis

An important mental health concern arising from the COVID-19 pandemic was the loneliness resulting from the lockdown measures taken by many countries due to the outbreak. Thus, loneliness needs to be studied in detail for intervention purposes. The UCLA Loneliness Scale is the most widely used measure of loneliness. However, the 20-item measure has some drawbacks, including questionnaire fatigue, especially when used with other measures, and a lack of accurate responses to all items. In this study, we evaluated two short forms of the UCLA Loneliness Scale used in other studies (UCLA-10 and UCLA-8), as well as a five-item version developed in this study, through the classical test theory and Mokken analysis with two different sample groups (teachers, N = 337; students, N = 355). The results of the Mokken analysis and the confirmatory factor analysis showed that the UCLA-10 and UCLA-5 were unidimensional and exhibited reliability and convergent validity. In contrast, the UCLA-8 was multidimensional and violated several assumptions of monotonicity and invariant item-ordering, although it showed satisfactory reliability and concurrent validity. All three versions of the scale correlated significantly with other indices of psychological well-being regarding concurrent validity, and the correlation coefficients were comparable to the coefficients of the 20-item version with the same variables. All three versions of the loneliness scale also demonstrated satisfactory convergent and discriminant validity. The results indicated that the UCLA-10 and UCLA-5 can be used as reliable and valid measures when the use of the long version of the UCLA Loneliness Scale is unsuitable.

Initial Development and Validation of the Measures of Stressors and Supports for Teachers (MOST)

The creation of psychometrically sound assessments of teacher well-being is critical given the alarmingly high rates of teacher burnout reported among U.S. educators. The present study sought to address this need by developing the Measures of Stressors and Supports for Teachers (MOST), a teacher-report questionnaire designed to assess ecological and psychological factors that affect teachers’ professional well-being. To assess structural validity, the MOST was administered to a sample of K–12 educators ( N = 218). Methods outlined in Classical Test Theory and exploratory factor analysis were conducted to examine items and assess the factor structure of the MOST. Factor analytic findings yielded a 40-item, nine-factor structure (Parents, Colleagues, School Leadership and Belonging, Classroom Students, Students With Disabilities, Time Pressure, Professional Development, Safety, and Emotional State). Implications for further validation and use of the MOST are discussed.

Developing Classroom Assessment Tool using Learning Management System-based Computerized Adaptive Test in Vocational High Schools

Computers have taken on a large role in education, including testing and evaluation. Traditional tests that aren't comprehensive and don't distinguish between students' beginning talents lead to measurement findings that aren't representative of their true abilities. This study aims to develop and test assessment tool eligibility class that is used as an LMS-based adaptive. This type of research includes development research. The respondents of this study were experts who assessed the validation and students of SMK Electrical Power Installation Engineering Expertise Competencies. The data analysis technique used item response theory, classical test theory and descriptive statistics. Item analysis using the Rasch Model showed 10 items were not fit and the remaining 50 items were fit. Classical test theory analysis items with less validity there are 0 items, moderate there are 55 items, and high there are 5 items, with an Alpha reliability of 0.934. The attitude questionnaire developed consists of 8 items. There are 0 items with less validity, 6 items being moderate, and 2 items high, with an Alpha reliability of 0.731. The developed observation guide contains 16 observations. Items with less validity have 0 items, while there are 15 items, and 1 item high. The use of CAT-assisted learning management systems on research tools greatly facilitates teachers in conducting research accurately and practically.

Impact of the Appalachian Career Training in Oncology (ACTION) Program on Undergraduate Participants.

Kentucky ranks first in the nation in cancer incidence and mortality. The Appalachian region of the state experiences the highest cancer disparities due to inequities in many social determinants of health. As a strategy for addressing cancer and education disparities in the region, the Appalachian Career Training in Oncology (ACTION) Program at the University of Kentucky Markey Cancer Center engaged 16 Appalachian-native undergraduate student participants annually in cancer-related activities. Students were recruited on an annual basis for the 2-year ACTION program. Entry, mid-point, and exit surveys were administered to participants. Classical test theory and exploratory factor analysis were used to examine the instruments used for program evaluation, whereas repeated measures ANOVA, paired t-tests, chi-squared, and post hoc analyses were used to analyze 6years of survey data. There was a significant increase in participants' understanding of cancer-related topics among the entry, midpoint, and exit surveys (p < .001) and a significant increase in comfort with research, clinical, and outreach activities between entry and midpoint and entry and exit (p < .001), but not between midpoint and exit. With respect to research skills and perceptions of the program, increases in mean scores were observed between midpoint and exit, but these increases were not statistically significant (p = .167, p = 223, respectively). ACTION increased participants' understanding of cancer-related topics; comfort with research, clinical, and outreach activities; and research skills. These data suggest that ACTION has a significant impact on participants.

Psychometric and Clinical Evaluation of the Clinician (VQIDS-C5) and Self-Report (VQIDS-SR5) Versions of the Very Quick Inventory of Depressive Symptoms.

PurposeEvaluate the psychometric properties of the 5-item Very Quick Inventory of Depressive Symptomatology self-report and clinician-rated versions (VQIDS-SR5/VQIDS-C5), compare their relative performance, create crosswalks between their total scores and other accepted depressive symptom ratings, and define clinically relevant depressive symptom severity thresholds and categorical outcomes for both versions.Patients and MethodsThe Sequenced Treatment Alternatives to Relieve Depression trial obtained baseline and exit 17-item Hamilton Rating Scale for Depression (HRSD17) and 30-item Inventory of Depressive Symptomatology – Clinician-rated scores, and baseline and visit-wise QIDS-SR16 and QIDS-C16 ratings from the first treatment step (citalopram). The VQIDS-C5 and the VQIDS-SR5 items (sad mood, self-outlook, involvement, fatigue, psychomotor slowing) (each rated 0–3), extracted from the corresponding 16-item ratings, were selected to best reflect the 6-item HRSD (HRSD6) (exclusive of anxiety). Classical Test Theory (CTT) and Item-Response Theory (IRT) analyses assessed psychometric features. IRT analyses produced total score crosswalks between the VQIDS5, QIDS-C16, QIDS-SR16 and HRSD6. Clinically relevant VQIDS symptom severity thresholds and treatment outcomes were estimated based on cross-walks from the parent QIDS16 ratings.ResultsBoth VQIDS versions were unifactorial with acceptable internal consistencies (Cronbach’s alphas >0.80), item-total correlations (0.57–0.74) by CCT, and strong IRT item performance. Based on QIDS16 severity thresholds (none 0–5; mild 6–10; moderate 11–15; severe 16–20; and very severe 21–27), comparable thresholds were 0–2; 3–5; 6–9; 9–12; and >12 for VQIDS-C5, and 0–2; 2–5; 6–8; 9–12; and >12 for VQIDS-SR5. Kappa values were acceptable in comparing categories of outcomes (eg, no benefit, remission, etc) based on VQIDS and corresponding QIDS categories.ConclusionThe VQIDS-C5 and VQIDS-SR5 assess selected core depressive symptoms with psychometrically acceptable properties. Theelf-report and clinician-rated versions provide virtually identical information, symptom severity thresholds and symptom change categories. Both are as sensitive to change as the corresponding QIDS16, making them suitable for use in busy practices.