Computerized Adaptive Testing Research Articles

Validation of the cognitive section of the Penn computerized adaptive test for neurocognitive and clinical psychopathology assessment (CAT-CCNB)

BackgroundThe Penn Computerized Neurocognitive Battery is an efficient tool for assessing brain-behavior domains, and its efficiency was augmented via computerized adaptive testing (CAT). This battery requires validation in a separate sample to establish psychometric properties. MethodsIn a mixed community/clinical sample of N = 307 18-to-35-year-olds, we tested the relationships of the CAT tests with the full-form tests. We compared discriminability among recruitment groups (psychosis, mood, control) and examined how their scores relate to demographics. CAT-Full relationships were evaluated based on a minimum inter-test correlation of 0.70 or an inter-test correlation within at least 0.10 of the full-form correlation with a previous administration of the full battery. Differences in criterion relationships were tested via mixed models. ResultsMost tests (15/17) met the minimum criteria for replacing the full-form with the updated CAT version (mean r = 0.67; range = 0.53–0.80) when compared to relationships of the full-forms with previous administrations of the full-forms (mean r = 0.68; range = 0.50–0.85). Most (16/17) CAT-based relationships with diagnostics and other validity criteria were indistinguishable (interaction p > 0.05) from their full-form counterparts. ConclusionsThe updated CNB shows psychometric properties acceptable for research. The full-forms of some tests should be retained due to insufficient time savings to justify the loss in precision.

Continuum of measurement: Reviewing the advantages and disadvantages of self‐administered remote cognitive tests and their examiner administered alternatives

AbstractBackgroundFour in 10 older adults own a smartphone in their 60s and beyond (Pew Research; 2017). Correspondingly, there has been surging interest in using digital cognitive assessments remotely to inform clinical and research practice with older adults and more comparisons between self‐ and examiner‐administered tests are needed.MethodAs part of a larger study, 1120 adults completed a battery of remotely administered tests (Mobile Toolbox) and a subset of this sample completed examiner administered in‐person testing (NIH Toolbox Cognition Battery). Attention was given to making the sample reflective of the US 2020 Census during participant recruitment. Of the 1120 participants, the majority of the sample were female (57%) and Caucasian (72%) and had a mean age of 45 (SD = 21). In terms of education, equal percentages had high school (34%) or some college (34%).ResultCognitive tests of processing speed, language, executive function, attention, and episodic memory were administered via a trained examiner and correlates of these tests were self‐administered remotely via a smartphone. Using examples, we will show which aspects of cognitive assessment had the best correlations between remote self‐administration and face‐to‐face examination and which had lower correlations.ConclusionFor geriatric individuals, remote assessments can be used to help over barriers for care, such as difficulty getting to the office location due to limited mobility or lack of transportation. Remote assessments reduce participant burden and expense and increase research accessibility for older adults. Capitalizing on the omnipresent nature of internet‐connected devices, this improves opportunities to remotely monitor other dimensions relevant to cognition using smartphone apps and wearable sensors. In addition to improving access to testing, digitally administered assessments dramatically improve some individual’s tolerance to testing with shorter tests that can be administered via computer adaptive testing (CAT). Despite these benefits, some aspects of the cognitive assessment cannot be adequately replicated remotely and thus yield lower correlations to their examiner‐administered alternatives. Clinical and research implications are discussed.

PCR71 Using Schedule for the Evaluation of Individual Quality of Life (SEIQoL) and Computer Adapted Testing (CAT) in Current Practices

PCR71 Using Schedule for the Evaluation of Individual Quality of Life (SEIQoL) and Computer Adapted Testing (CAT) in Current Practices

Continuum of measurement: Reviewing the advantages and disadvantages of self‐administered remote cognitive tests and their examiner administered alternatives

AbstractBackgroundFour in 10 older adults own a smartphone in their 60s and beyond (Pew Research; 2017). Correspondingly, there has been surging interest in using digital cognitive assessments remotely to inform clinical and research practice with older adults and more comparisons between self‐ and examiner‐administered tests are needed.MethodAs part of a larger study, 1120 adults completed a battery of remotely administered tests (Mobile Toolbox) and a subset of this sample completed examiner administered in‐person testing (NIH Toolbox Cognition Battery). Attention was given to making the sample reflective of the US 2020 Census during participant recruitment. Of the 1120 participants, the majority of the sample were female (57%) and Caucasian (72%) and had a mean age of 45 (SD = 21). In terms of education, equal percentages had high school (34%) or some college (34%).ResultCognitive tests of processing speed, language, executive function, attention, and episodic memory were administered via a trained examiner and correlates of these tests were self‐administered remotely via a smartphone. Using examples, we will show which aspects of cognitive assessment had the best correlations between remote self‐administration and face‐to‐face examination and which had lower correlations.ConclusionFor geriatric individuals, remote assessments can be used to help over barriers for care, such as difficulty getting to the office location due to limited mobility or lack of transportation. Remote assessments reduce participant burden and expense and increase research accessibility for older adults. Capitalizing on the omnipresent nature of internet‐connected devices, this improves opportunities to remotely monitor other dimensions relevant to cognition using smartphone apps and wearable sensors. In addition to improving access to testing, digitally administered assessments dramatically improve some individual’s tolerance to testing with shorter tests that can be administered via computer adaptive testing (CAT). Despite these benefits, some aspects of the cognitive assessment cannot be adequately replicated remotely and thus yield lower correlations to their examiner‐administered alternatives. Clinical and research implications are discussed.

Students’ perceptions of Computerised Adaptive Testing in higher education

The COVID-19 pandemic has forced higher education institutions (HEIs) to re-look at their assessment strategy as learning, development, and engagement move more fluidly into the online arena. The purpose of this research was to investigate students' academic and personal perceptions of computer adaptive testing (CAT) in higher education to understand students’ confidence in adopting CAT. Using a quantitative descriptive research design, an online questionnaire was administered to students at private and public HEIs in South Africa, with 600 respondents. The study found that the students were comfortable engaging in online learning and expressed positive perceptions of adopting CAT, with most respondents recommending it for implementation. Students believe that CAT allowed for more productive interaction with material which meets their needs and learning preferences without feeling overburdened. The findings of the study provide HEIs with valuable information on key managerial implications to ensure the successful adoption and implementation of CAT.

Social Networks, the COVID-19 Pandemic, and Emerging Adults' Mental Health: Resiliency Through Social Bonding and Cohesion.

Objectives. To assess how personal social network characteristics moderated mental health declines during the COVID-19 pandemic in emerging adults compared with other age groups. Methods. The Person to Person Health Interview Study, a representative, probability-based cohort study (n = 2485) in Indiana, collected data through face-to-face (baseline) and phone (follow-up) interviews before and during the pandemic. We used survey-weighted growth curve models to examine network effects on computer-adaptive testing measures of depression and anxiety severity. Results. Respondents reported significantly increased depression and anxiety in 2021, which returned almost to baseline levels for most age groups by 2022 (P < .001). Stronger ties to others and more interconnected ties were significantly associated with lower depression (B = -0.112 [P < .05]; B = -0.086 [P < .001]) and anxiety (B = -0.101 [P < .05]; B = -0.063 [P < .01]) severity across the pandemic. Interaction models revealed disproportionate protective effects of network characteristics on depression (B = -0.456 [P < .001]; B = -0.268 [P < .001]) and anxiety (B = -0.388 [P < .001]; B = -0.284 [P < .001]) for emerging adults. Conclusions. Cohesive and affectively strong personal networks promote resiliency to common mental health challenges during periods of crisis, particularly for emerging adults whose social roles and relationships were disrupted during a critical period of development. (Am J Public Health. 2024;114(S3):S258-S267. https://doi.org/10.2105/AJPH.2023.307426).

Patient-Reported Outcome Measures for Patients With CKD: The Case for Patient-Reported Outcomes Measurement Information System (PROMIS) Tools

Chronic kidney disease (CKD), kidney failure, and kidney replacement therapies are associated with high symptom burden and impaired health-related quality of life (HRQOL). Symptoms change with disease progression or transition between treatment modalities and frequently go unreported and unmanaged. Tools that reliably monitor symptoms may improve the management of patients with CKD. Patient-reported outcome measures (PROMs) assess symptom severity; physical, psychological, social, and cognitive functioning; treatment-related side effects; and HRQOL. Systematic use of PROMs can improve patient-provider communication, patient satisfaction, clinical outcomes, and HRQOL. Potential barriers to their use include a lack of engagement, response burden, and limited guidance about PROM collection, score interpretation, and workflow integration. Well-defined, acceptable, and effective clinical response pathways are essential for implementing PROMs. PROMs developed by the Patient-Reported Outcomes Measurement Information System (PROMIS) address some challenges and may be suitable for clinical use among patients with CKD. PROMIS tools assess multiple patient-valued, clinically actionable symptoms and functions. They can be administered as fixed-length, customized short forms or computer adaptive tests, offering precise measurement across a range of symptom severities or function levels, tailored questions to individuals, and reduced question burden. Here we provide an overview of the potential use of PROMs in CKD care, with a focus on PROMIS.

Developing of Computerized Adaptive Test (CAT) Based on a Learning Management System in Mathematics Final Exam for Junior High School

Purpose: The research aimed to develop a valid, feasible, and efficient computerized adaptive test (CAT) to measure students’ abilities on the end-of-semester exams. Design/Approach/Methods: This research was a 4D research and development with four stages, namely, definition, design, development, and dissemination. Stratified random sampling techniques were used in the study by randomly sampling based on high, moderate, and low levels of academic achievement ability. The study sample was 637 junior high school students in Yogyakarta. Finding: The results showed that the developed CAT program was considered very good and feasible to use. The validity of the contents was obtained by 200 valid questions, and in the characteristics of the item, all items were declared to fit with the Rasch model. All problem items with item difficulty index were relatively good. The reliability of the item estimate was in the very good category while the reliability of the case estimate was in the good category. The lowest and highest ability test results obtained were in the range of 1.1 to 9.85. The ability of test takers was diverse and had scattered distributions. The feasibility of using CAT media was obtained from teachers and students indicating that CAT media was in a very feasible category to use.

4 Continuum of Measurement: Reviewing the Advantages and Disadvantages of Self-Administered Remote Cognitive Tests and Their Examiner Administered Alternatives

Objective:To describe advantages and disadvantages of using digital assessments remotely and in-person to inform clinical and research practice.Participants and Methods:As part of a larger study,1120 adults completed a battery of remotely administered tests (Mobile Toolbox) and a subset of this sample completed examiner administered in-person testing (NIH Toolbox® Cognition Battery). Attention was given to making the sample reflective of the US 2020 Census during participant recruitment. Of the 1120 participants, the majority of the sample were female (57%) and Caucasian (72%) and had a mean age of 45 (SD = 21). In terms of education, equal percentages had high school (34%) or some college (34%).Results:NIH Toolbox cognitive tests of processing speed, language, executive function, attention, and episodic memory were administered via a trained examiner and correlates of these tests were self-administered remotely via a smartphone. Using examples, we will show which aspects of cognitive assessment had the best correlations between remote self-administration and face-to-face examination and which had lower correlations.Conclusions:Digital remote assessments can help overcome barriers by enabling repeated testing in naturalistic conditions, reducing participant burden and expense, and increasing research accessibility for populations currently under-represented. Moreover, the ubiquity of internet-connected devices vastly increases opportunities to remotely monitor other dimensions relevant to cognition using smartphone apps and wearable sensors. In addition to improving access to testing, digitally administered assessments dramatically improve some individual’s tolerance to testing with shorter tests that can be administered via computer adaptive testing (CAT). Despite these benefits, some aspects of the cognitive assessment cannot be adequately replicated remotely and thus yield lower correlations to their examiner-administered alternatives. Clinical and research implications are discussed.

3 Stricker Learning Span criterion validity: remote self-administration of a computer adaptive word list memory test shows similar ability to differentiate PET-defined biomarker groups as in-person Rey Auditory Verbal Learning Test performance in cognitively unimpaired individuals on the Alzheimer’s continuum

Objective:The Stricker Learning Span (SLS) is a computer-adaptive word list memory test specifically designed for remote assessment and self-administration on a web-based multi-device platform (Mayo Test Drive). Given recent evidence suggesting the prominence of learning impairment in preclinical Alzheimer’s disease (AD), the SLS places greater emphasis on learning than delayed memory compared to traditional word list memory tests (see Stricker et al., Neuropsychology in press for review and test details). The primary study aim was to establish criterion validity of the SLS by comparing the ability of the remotely-administered SLS and inperson administered Rey Auditory Verbal Learning Test (AVLT) to differentiate biomarkerdefined groups in cognitively unimpaired (CU) individuals on the Alzheimer’s continuum.Participants and Methods:Mayo Clinic Study of Aging CU participants (N=319; mean age=71, SD=11; mean education=16, SD=2; 47% female) completed a brief remote cognitive assessment (∼0.5 months from in-person visit). Brain amyloid and brain tau PET scans were available within 3 years. Overlapping groups were formed for 1) those on the Alzheimer’s disease (AD) continuum (A+, n=110) or not (A-, n=209), and for 2) those with biological AD (A+T+, n=43) vs no evidence of AD pathology (A-T-, n=181). Primary neuropsychological outcome variables were sum of trials for both the SLS and AVLT. Secondary outcome variables examined comparability of learning (1-5 total) and delay performances. Linear model ANOVAs were used to investigate biomarker subgroup differences and Hedge’s G effect sizes were derived, with and without adjusting for demographic variables (age, education, sex).Results:Both SLS and AVLT performances were worse in the biomarker positive relative to biomarker negative groups (unadjusted p’s&lt;.05). Because biomarker positive groups were significantly older than biomarker negative groups, group differences were attenuated after adjusting for demographic variables, but SLS remained significant for A+ vs A- and for A+T+ vs A-T- comparisons (adjusted p’s&lt;.05) and AVLT approached significance (p’s .05-.10). The effect sizes for the SLS were slightly better (qualitatively, no statistical comparison) for separating biomarker-defined CU groups in comparison to AVLT. For A+ vs A- and A+T+ vs A-T- comparisons, unadjusted effect sizes for SLS were -0.53 and -0.81 and for AVLT were -0.47 and -0.61, respectively; adjusted effect sizes for SLS were -0.25 and -0.42 and for AVLT were -0.19 and -0.26, respectively. In secondary analyses, learning and delay variables were similar in terms of ability to separate biomarker groups. For example, unadjusted effect sizes for SLS learning (-.80) was similar to SLS delay (.76), and AVLT learning (-.58) was similar to AVLT 30-minute delay (-.55) for the A+T+ vs AT- comparison.Conclusions:Remotely administered SLS performed similarly to the in-person-administered AVLT in its ability to separate biomarker-defined groups in CU individuals, providing evidence of criterion validity. The SLS showed significantly worse performance in A+ and A+T+ groups (relative to A- and A-T-groups) in this CU sample after demographic adjustment, suggesting potential sensitivity to detecting transitional cognitive decline in preclinical AD. Measures emphasizing learning should be given equal consideration as measures of delayed memory in AD-focused studies, particularly in the preclinical phase.

Developing Computerized Adaptive Testing for a National Health Professionals Exam: An Attempt from Psychometric Simulations.

The accurate assessment of health professionals' competence is critical for ensuring public health safety and quality of care. Computerized Adaptive Testing (CAT) based on the Item Response Theory (IRT) has the potential to improve measurement accuracy and reduce respondent burden. In this study, we conducted psychometric simulations to develop a CAT for evaluating the candidates' competence of health professionals. The initial CAT item bank was sourced from the Standardized Competence Test for Clinical Medicine Undergraduates (SCTCMU), a nationwide summative test in China, consisting of 300 multiple-choice items. We randomly selected response data from 2000 Chinese clinical medicine undergraduates for analysis. Two types of analyses were performed: first, evaluating the psychometric properties of all items to meet the requirements of CAT; and second, conducting multiple CAT simulations using both simulated and real response data. The final CAT item bank consisted of 121 items, for which item parameters were calculated using a two-parameter logistic model (2PLM). The CAT simulations, based on both simulated and real data, revealed sufficient marginal reliability (coefficient of marginal reliability above 0.750) and criterion-related validity (Pearson's correlations between CAT scores and aggregate scores of the SCTCMU exceeding 0.850). In national-level medical education assessment, there is an increasing need for concise yet valid evaluations of candidates' competence of health professionals. The CAT developed in this study demonstrated satisfactory reliability and validity, offering a more efficient assessment of candidates' competence of health professionals. The psychometric properties of the CAT could lead to shorter test durations, reduced information loss, and a decreased testing burden for participants.

Equipercentile equating of scores from common patient-reported outcome measures of physical function in patients with cancer

ObjectiveTo provide equipercentile equating of physical function (PF) scores from frequently used patient-reported outcome measures (PROMs) in cancer patients to facilitate data pooling and comparisons. Study Design and SettingAdult cancer patients from five European countries completed the European Organization for Research and Treatment of Cancer (EORTC) computer adaptive test (CAT) Core, EORTC Quality of Life Questionnaire Version 3.0 (QLQ-C30), Functional Assessment of Cancer Therapy - General (FACT-G), 36-item Short Form Health Survey (SF-36), and the Patient-Reported Outcomes Measurement Information System (PROMIS) Physical Function 20a short form. The R package “equate” was used to establish conversion tables of PF scores on those measures with a bivariate rank correlation of at least 0.75. ResultsIn total, 953 patients with cancer (mean age 58.9 years, 54.7% men) participated. Bivariate rank correlations between PF scores from the EORTC CAT Core, EORTC QLQ-C30, SF-36, and PROMIS were all above 0.85, but below 0.69 for the FACT-G. Conversion tables were established for all measures but the FACT-G. These tables indicate which score from one PROM best matches the score from another PROM and provide standard errors of converted scores. ConclusionOur analysis indicates that linking of PF scores from both EORTC measures (CAT and QLQ-C30) with PROMIS and SF-36 is possible, whereas the physical domain of the FACT-G seems to be different. The established conversion tables may be used for comparing results or pooling data from clinical studies using different PROMs.

Patient-Reported Outcome Measurement Information System Depression and Anxiety in Elective Knee Surgery Patients.

Mental health has been shown to play an important role in patient-reported outcomes (PRO); however, there is a general lack of literature describing patient-reported outcome measurement information system (PROMIS) depression and anxiety computer adaptive tests in elective knee surgery patients. The purpose of our study was to assess the prevalence of depression and anxiety symptoms before and after elective knee surgery and to determine whether these symptoms influence postoperative functional outcomes. An institutional review board-approved prospective orthopaedic registry was retrospectively queried for patients undergoing elective knee surgery from June 2015 to November 2018. Electronic surveys collecting patient demographic information and PROs were administered pre- and postoperatively. Of the 663 patients that completed baseline questionnaires, 466 completed 2-year follow-up (70.3%). PROs included PROMIS depression, PROMIS anxiety, International Knee Documentation Committee Subjective Knee Form (IKDC), and PROMIS physical function (PF). Wilcoxon rank sum and Spearman's rank order correlation were utilized to determine associations between variables. Multivariable analysis was used to control for confounding variables. Average PROMIS depression and anxiety scores significantly improved 2 years after surgery. PROMIS depression and anxiety scores significantly correlated with each other. PROMIS depression and anxiety scores significantly correlated with PROMIS PF and IKDC scores. After controlling for confounders on multivariable analysis, worse 2-year PROMIS anxiety was predictive of less functional improvement and worse 2-year PF and IKDC, while worse 2-year PROMIS depression was predictive of less improvement in IKDC. This study confirms the important relationship between mental health and functional outcomes. Given that psychiatric comorbidities are potentially modifiable with treatment, proper recognition could potentially lead to better orthopaedic outcomes. In addition, the prevalence of depression and anxiety symptoms postoperatively, as documented by PROMIS computer adaptive tests, may act as a barrier to achieving optimal functional outcomes after elective knee surgery. LEVEL OF EVIDENCE: Level III.

Controlling the Minimum Item Exposure Rate in Computerized Adaptive Testing: A Two-Stage Sympson-Hetter Procedure.

Computerized adaptive testing (CAT) can improve test efficiency, but it also causes the problem of unbalanced item usage within a pool. The effect of uneven item exposure rates can not only induce a test security problem due to overexposed items but also raise economic concerns about item pool development due to underexposed items. Therefore, this study proposes a two-stage Sympson-Hetter (TSH) method to enhance balanced item pool utilization by simultaneously controlling the minimum and maximum item exposure rates. The TSH method divides CAT into two stages. While the item exposure rates are controlled above a prespecified level (e.g., rmin) in the first stage to increase the exposure rates of the underexposed items, they are controlled below another prespecified level (e.g., rmax) in the second stage to prevent items from overexposure. To reduce the effect on trait estimation, TSH only administers a minimum sufficient number of underexposed items that are generally less discriminating in the first stage of CAT. The simulation study results indicate that the TSH method can effectively improve item pool usage without clearly compromising trait estimation precision in most conditions while maintaining the required level of test security.

Communicative participation outcomes in individuals with Parkinson's disease receiving standard care speech-language therapy services in community settings.

The Communicative Participation Item Bank (CPIB) is a patient-reported outcome measure (PROM) designed to measure the extent of interference, or difficulty, experienced by adults with communication disorders participating in their day-to-day communication activities. To date, there is limited evidence regarding sensitivity of the CPIB for capturing change with intervention in people with Parkinson's disease (PwPD). The purpose of this study was to examine the following measurement properties of the CPIB in PwPD who received community-based, standard care, speech-language therapy focusing on motor speech concerns: Change over time between treatment and observation groups, comparison to patient-defined ideal and satisfactory targets, comparison of static short form to computerised adaptive testing (CAT), comparison of self to proxy-rated scores, and comparison to other common PROMs. Forty-six PwPD (20 treatment/26 observation) completed data collection upon enrolment (pre-treatment) and 6 months later. In addition to the CPIB, PROMs included the Voice Handicap Index 10-item short form (VHI-10), PROMIS Global Health-Related Quality of Life, Levels of Speech Usage, self-rated speech severity, and Patient Health Questionnaire-9 (PHQ-9). Participants also engaged in qualitative interviews. Forty-four family members completed proxy CPIB ratings. There were no significant differences between treatment and observation groups on the CPIB pre-treatment, but there were significant differences post-treatment. The differences appeared to be largely due to significant gains in the treatment group. No participants reached their ideal CPIB target, and few reached their satisfactory target. Static CPIB short form and CAT scores were not significantly different, with an average of five CAT items administered per participant. Overall group similarities between patient and proxy scores may have obscured wide variability across individual patient-proxy pairs. Associations between CPIB and VHI-10, health-related quality of life, self-reported speech severity, and depression ranged from weak to moderate. The CPIB appears to be sensitive to capturing change with intervention, and similar results are obtained with the static short form and CAT formats. One clinical caution is that even with gains observed in the treatment group, no participants obtained their ideal communicative participation goals, and few obtained a satisfactory level of communicative participation. Thus, while current interventions are beneficial, they may not meet the full range of clients' communication needs. While responding to the CPIB through a proxy rater may be feasible, caution is warranted due to concerns about maintaining the autonomy of PwPD. What is already known on this subject The communication disorders associated with Parkinson's disease (PD) can have a negative impact on quality of life and life participation as measured by patient (or person)-reported outcome measures (PROMs). The Communicative Participation Item Bank (CPIB) is one PROM available to use with adults with communication disorders. However, little is known about whether the CPIB captures changes in communicative participation as a result of standard care treatment for people with Parkinson's disease (PwPD). Use of computerised adaptive testing (CAT), proxy report and comparison to targeted participation outcomes have not been explored. What this study adds to existing knowledge As a result of this study, we know that the CPIB captured differences between treatment and observation groups after community-based, standard care speech therapy intervention focusing on motor speech production in PwPD. Static short form and CAT scores did not differ significantly, so the CAT option provides better efficiency requiring, on average, five items to administer compared to the 10-item short form. Proxy and PwPD scores did not differ as a group, but wide variability was noted. What are the potential or actual clinical implications of this work? The CPIB may be a clinically sensitive instrument for capturing changes in communicative participation after treatment. No participants met their ideal CPIB target, and few reached their satisfactory target, suggesting that while current interventions contribute to gains in communicative participation, there are still unmet needs that may call for support and interventions addressing the more complex array of factors affecting communicative participation outcomes for PwPD.

Two Statistics for Measuring the Score Comparability of Computerized Adaptive Tests.

This study introduces two new statistics for measuring the score comparability of computerized adaptive tests (CATs) based on comparing conditional standard errors of measurement (CSEMs) for examinees that achieved the same scale scores. One statistic is designed to evaluate score comparability of alternate CAT forms for individual scale scores, while the other statistic is designed to evaluate the overall score comparability of alternate CAT forms. The effectiveness of the new statistics is illustrated using data from grade 3 through 8 reading and math CATs. Results suggest that both CATs demonstrated reasonably high levels of score comparability, that score comparability was less at very high or low scores where few students score, and that using random samples with fewer students per grade did not have a big impact on score comparability. Results also suggested that score comparability was sometimes higher when the bottom 20% of scorers were used to calculate overall score comparability compared to all students. Additional discussion related to applying the statistics in different contexts is provided.

Development of a Computerized-Adaptive Test to Measure English Vocabulary Size with IRT

To speak English fluently like a native speaker, having a vocabulary of 20,000 words are required. In many universities in Indonesia, Vocabulary Size Test (VST) is still held with Paper-Based Test (PBT), so it is less effective and efficient. The aims of this study were 1) to develop Computerized-Adaptive Testing (CAT), which helps get students with a standardized English vocabulary size, and 2) to implement it in measuring vocabulary size. The approach in this study is Research and Development (RD) with product development procedures following the steps of Borg and Gall. The findings of this study are, 1) the CAT program has Item Response Theory (IRT) parameters that can change according to the test takers' answers, and the test will stop if the estimated ability of the test takers is known (Standard Error/SE close to 0.01), and 2) with the application of the CAT program it can be seen that the total vocabulary of English Language and Teaching Department students is still 20,000. Based on these findings, this study implies that for students' vocabulary mastery to be on par with native speakers' abilities, the English Language and Teaching Department needs to evaluate the learning process and determine a particular program to increase student vocabulary.

Psychometric properties of the Chinese version of the PROMIS-Cancer-Anxiety item bank assessed using a graded response model

ObjectiveThis study aimed to examine the psychometric properties of the Chinese version of the Patient-Reported Outcome Measurement Information System (PROMIS)-Cancer-Anxiety item bank using a graded response model in a sample of patients with cancer. MethodsA cross-sectional study was conducted and the Chinese version of the PROMIS-Cancer-Anxiety item bank was used to measure anxiety in patients with cancer. The unidimensional structure of the item bank was evaluated using principal component analysis. Residual correlations and the graphs of item mean scores conditional on the rest scores were examined to evaluate the local independence and monotonicity of the items, respectively. Item characteristics were described using item parameter estimates and item information. Operating characteristic curves (OCCs) and test information curve (TIC) were also plotted. Measurement invariance across age, gender, and education level was assessed to identify possible differential item functioning (DIF). ResultsA total of 1075 patients with cancer were enrolled. Under the assumptions of unidimensionality, local independence, and monotonicity, the discrimination parameters a ranged from 2.30 to 5.47, and the threshold parameters b ranged from b1 = −2.87 to b4 = 3.21 with proper intervals. Completely overlapped category curves were not observed among the OCCs of any items. Item information and TIC showed that the item bank had a wide measurement range. The DIFs for age, gender, and education level for all items were not remarkable. ConclusionsThe results supported using the Chinese version of the PROMIS-Cancer-Anxiety item bank to measure anxiety and develop a computerized adaptive testing (CAT) system for anxiety in patients with cancer.

Utilizing Real-Time Test Data to Solve Attenuation Paradox in Computerized Adaptive Testing to Enhance Optimal Design

To solve the attenuation paradox in computerized adaptive testing (CAT), this study proposes an item selection method, the integer programming approach based on real-time test data (IPRD), to improve test efficiency. The IPRD method turns information regarding the ability distribution of the population from real-time test data into feasible test constraints to reversely assembled shadow tests for item selection to prevent the attenuation paradox by integer programming. A simulation study was conducted to thoroughly investigate IPRD performance. The results indicate that the IPRD method can efficiently improve CAT performance in terms of the precision of trait estimation and satisfaction of all required test constraints, especially for conditions with stringent exposure control.

Application of the professional maturity scale as a computerized adaptive testing

This study has two main purposes. Firstly, to compare the different item selection methods and stopping rules used in Computerized Adaptive Testing (CAT) applications with simulative data generated based on the item parameters of the Vocational Maturity Scale. Secondly, to test the validity of CAT application scores. For the first purpose, simulative data produced based on Vocational Maturity Scale item parameters were analyzed under different item selection methods (Maximum Fisher Information [MFI],Maximum Likelihood Weighted Information [MLWI] Maximum Posterior Weighted Information [MPWI] Maximum Expected Information [MEI] Minimum Expected Posterior Variance [MEPV] Maximum Expected Posterior Weighted Information [MEPWI]) and stopping rules (Standard Error [SE]&amp;lt;0.30, SE&amp;lt;0.50, SE &amp;lt;0.70, Number of Item [NI]=10, NI=20) by calculating the average number of items, standard error averages, correlation coefficients, bias, and RMSE statistics. For all the conditions of the item selection methods, standard error averages, correlation coefficients, bias, and RMSE statistics showed similar results. When the average number of items is considered, MFI and SE&amp;lt;0.30 were found as most appropriate methods to be used in CAT application. For the second purpose of the study, the paper-pencil form of the Vocational Maturity scale and CAT version were administered to 33 students. A moderate, positive, and statistically significant relationship was found between the CAT application scores and the paper-pencil form scores on the vocational maturity scale. As a result, it can be said that the vocational maturity scale can be applied as a computerized adaptive test and can be used in career guidance processes.