Computerized Adaptive Testing Research Articles

Validation of Patient-Reported Outcome Measurement Information System for Detection of Posttraumatic Stress in Children and Adolescents Following Procedures for Acute Orthopaedic Trauma.

The efficiency and validity of the Patient-Reported Outcomes Measurement Information System (PROMIS) surveys were determined for pediatric orthopaedic trauma patients with posttraumatic stress disorder (PTSD) symptoms in a clinic setting. Prospective cohort study. Single-institution, Level I trauma center. All consecutive children aged 8-18 years undergoing procedures or surgery for orthopaedic trauma. The convergent, divergent, and discriminant validity of the PROMIS Anger and Anxiety computerized adaptive tests (CATs) were evaluated and compared with the previously validated Child PTSD Symptom Scale (CPSS). The efficiency in time to completion of the outcome measures was compared between the CPSS and PROMIS surveys. Cutoffs for increased likelihood of PTSD were established for the PROMIS questionnaires. A total of 233 subjects were included in this study (mean age 13.1 years with SD 2.8 years, 71% male). The majority (51%) of injuries were related to sports, and most (60%) involved the upper extremity. Of those included, 41 patients had high levels of PTSD symptoms on the CPSS (18%; 95% CI, 13.1-23.2%). The CPSS took 182 (interquartile range [IQR] 141-228) seconds versus 52 (IQR 36-84) and 52 (IQR 36-70) seconds for PROMIS Anger and Anxiety CATs, respectively. Convergent validity showed patient scores on both PROMIS instruments significantly correlated with CPSS scores (Anger: P < 0.001, r = 0.51; Anxiety: P < 0.001, r = 0.41). Neither PROMIS score correlated with University of California Los Angeles Activity Score (Anger: r = -0.26; Anxiety: r = -0.22), a functional outcome measure, demonstrating divergent validity. Both PROMIS instruments sufficiently discriminated across PTSD risk groups (Anger P < 0.001; Anxiety P < 0.001). A score of at least 53 on PROMIS Anger or at least 48 on PROMIS Anxiety indicated an increased likelihood of PTSD risk. PROMIS Anger and Anxiety CATs are efficient and valid for evaluating posttraumatic stress in children following orthopaedic trauma procedures. Diagnostic Level I. See Instructions for Authors for a complete description of levels of evidence.

Gross Motor Family Report: Refinement and evaluation of psychometric properties.

To refine the Gross Motor Family Report (GM-FR) using parents' input and to evaluate its psychometric properties. In this measurement study, 12 parents of children and adolescents with cerebral palsy (CP), aged 2 to 18 years, classified in all levels of the Gross Motor Function Classification System (GMFCS), were interviewed about their experience completing the GM-FR (content validity). Parents' feedback was used to refine the measure which was then completed by 146 families to evaluate internal consistency, and discriminative and concurrent validity. Forty-six parents completed the GM-FR again, 7 to 30 days later, to evaluate test-retest reliability. GM-FR scoring, pictures, descriptions, and the total number of items were revised based on parents' feedback. The GM-FR version 2.0 demonstrated high internal consistency (Cronbach's α = 0.99), no floor/ceiling effects, and excellent test-retest reliability (intraclass correlation coefficient = 0.99). GM-FR scores discriminated between GMFCS levels (p < 0.05) and were strongly negatively correlated with GMFCS level (r = -0.92; p < 0.001). GM-FR scores correlated positively and strongly with the Gross Motor Function Measure-66 (r = 0.94; p < 0.001) and the Pediatric Evaluation of Disability Inventory - Computer Adaptive Test mobility domain (r = 0.93; p < 0.001). Active participation of families in the GM-FR's development facilitated creation of a family-friendly instrument. This study provides strong evidence of reliability and validity to support GM-FR use in clinical practice and research for assessing gross motor performance of children and adolescents with CP.

Long-term efficacy, safety, and patient-reported outcomes of apitegromab in patients with spinal muscular atrophy: results from the 36-month TOPAZ study.

At 12 months in the phase 2 TOPAZ study, treatment with apitegromab was associated with both an improved motor function in patients with Type 2 or 3 spinal muscular atrophy (SMA) and with a favorable safety profile. This manuscript reports the extended efficacy and safety in the nonambulatory group of the TOPAZ study at 36 months. Patients who completed the primary study (NCT03921528) could enroll in an open-label extension, during which patients received apitegromab 20 mg/kg by intravenous infusion every 4 weeks. Patients were assessed periodically via the Hammersmith Functional Motor Scale-Expanded (HFMSE), Revised Upper Limb Module (RULM), World Health Organization (WHO) motor development milestones, Pediatric Evaluation of Disability Inventory Computer Adaptive Test (PEDI-CAT) Daily Activities and Mobility domains, and Patient-Reported Outcomes Measurement Information System (PROMIS) Fatigue questionnaire. Of the 58 patients enrolled in TOPAZ, 35 were nonambulatory (mean age 7.3 years). The mean change at 36 months in HFMSE score from baseline was +4.0 (standard deviation [SD]: 7.54), and + 2.4 (3.24) for RULM score (excluding n = 7 after scoliosis surgery). Caregiver-reported outcomes (PEDI-CAT and PROMIS Fatigue) showed improvements from baseline over 36 months. In addition, most patients (28/32) improved or maintained WHO motor milestones achieved at baseline. The most frequently reported treatment-emergent adverse events were pyrexia (48.6%), nasopharyngitis (45.7%), COVID-19 infection (40.0%), vomiting (40.0%), and upper respiratory tract infection (31.4%). The benefit of apitegromab treatment observed at 12 months was sustained at 36 months with no new safety findings.

Middle School Students' Conceptualizations and Reasoning about the Fairness of Math Tests

Assessment is central to teaching and learning, and recently there has been a substantive shift from paper-and-pencil assessments towards technology delivered assessments such as computer-adaptive tests. Fairness is an important aspect of the assessment process, including design, administration, test-score interpretation, and data utility. The Universal Design for Learning (UDL) guidelines can inform assessment development to promote fairness; however, it is not explicitly clear how UDL and fairness may be linked through students’ conceptualizations of assessment fairness. This phenomenological study explores how middle grades students conceptualize and reason about the fairness of mathematics tests, including paper-and-pencil and technology-delivered assessments. Findings indicate that (a) students conceptualize fairness through unique notions related to educational opportunities and (b) students’ reason about fairness non-linearly. Implications of this study have potential to inform test developers and users about aspects of test fairness, as well as educators data usage from fixed-form, paper-and-pencil tests, and computer-adaptive, technology-delivered tests.

Comparison of real data and simulated data analysis of a stopping rule based on the standard error of measurement in computerized adaptive testing for medical examinations in Korea: a psychometric study.

This study aimed to compare and evaluate the efficiency and accuracy of computerized adaptive testing (CAT) under 2 stopping rules (standard error of measurement [SEM]=0.3 and 0.25) using both real and simulated data in medical examinations in Korea. This study employed post-hoc simulation and real data analysis to explore the optimal stopping rule for CAT in medical examinations. The real data were obtained from the responses of 3rd-year medical students during examinations in 2020 at Hallym University College of Medicine. Simulated data were generated using estimated parameters from a real item bank in R. Outcome variables included the number of examinees’ passing or failing with SEM values of 0.25 and 0.30, the number of items administered, and the correlation. The consistency of real CAT result was evaluated by examining consistency of pass or fail based on a cut score of 0.0. The efficiency of all CAT designs was assessed by comparing the average number of items administered under both stopping rules. Both SEM 0.25 and SEM 0.30 provided a good balance between accuracy and efficiency in CAT. The real data showed minimal differences in pass/ fail outcomes between the 2 SEM conditions, with a high correlation (r=0.99) between ability estimates. The simulation results confirmed these findings, indicating similar average item numbers between real and simulated data. The findings suggest that both SEM 0.25 and 0.30 are effective termination criteria in the context of the Rasch model, balancing accuracy and efficiency in CAT.

Asking the "Right" Questions about Financial Hardship: Using Cognitive Interviews with Adolescents and Young Adults with Cancer and Their Caregivers to Inform Measure Development.

Purpose: Financial hardship as a result of cancer treatment can have a significant and lasting negative impact on adolescents and young adults (AYAs) and their families. To address a lack of developmentally informed and psychometrically sound measures of financial hardship for AYAs and their caregivers, we used rigorous measurement development methods recommended by the National Institutes of Health's Patient-Reported Outcomes Measurement Information System® (PROMIS®) to determine comprehensibility and relevance of measure content. Methods: Our multi-step approach involved item identification, refinement, and generation; translatability and reading level review; and cognitive interviews. A purposive sample of 25 AYAs and 10 caregivers participated, ensuring representation across age, education, gender, race/ethnicity, and cancer type. Results: Fifty patient-reported and caregiver-reported items were developed across material, psychosocial, and behavioral subdomains of financial hardship. Translatability and reading level reviews resulted in 22 patient-reported and 25 caregiver-reported items being rewritten. Eighty-eight percent of patients and all caregivers described the items as easy to answer. Younger AYAs (15 to 25 years of age) were more likely to say the items were less relevant for them. Forty-six patient-reported and 48 caregiver-reported items were recommended for further testing. Conclusion: This study is the first to use in-depth qualitative methods to center AYA patient and caregiver experiences in the creation of new measures of financial hardship. Data support the comprehensibility and content validity of these preliminary item banks. Future large-scale, quantitative testing will lead to additional refinements and support the use of short forms and computer-adaptive testing for a diverse sample of AYAs and their caregivers.

Assessing the Potentials of Compurized Adaptive Testing to Enhance Mathematics and Science Student’t Achievement in Secondary Schools

This study focuses on Computerized Adaptive Testing (CAT), designed to provide Mathematics and Basic Science teachers in secondary schools with an innovative method of assessment, that can enhance student’s achievement. A descriptive survey design was adopted for the study. Three research questions and two null hypotheses were formulated to guide the study. Using purposive sampling technique, a sample size of four hundred (400) Junior secondary three (JSS3) students and four private schools with computer laboratories in Akwa Ibom State were selected for the study. Each group had 100 members with mixed Mathematical and Basic Science competencies of high, medium, and low. The instrument for data collection was a Mathematics and Basic Science Computerized Adaptive Test (MABSCAT) developed by the researchers, tested, and modified after trial testing.Descriptive statistics, one-way ANOVA was used to analysed the data. Results indicated that: test lengths, testing times, and Mathematical and Basic Science competencies were significantly different at p=0.05 among four groups of stopping criteria (SEE ≤ 0.20, SEE ≤ 0.30, SEE ≤ 0.40 , and SEEm- SEEm-1 ≤ 0.005); test lengths and testing times were significantly different at p=0.05 among the three groups of Mathematical and Basic Science competencies (low, moderately high, and high); and there were 72.25 %, 16.75%, and 8% of the students having a moderately high, low, and high Mathematics and Basic Science achievement respectively. It was recommended among others that, Government should establish item banks, CAT should be introduced in assessment of learning for the actual abilities of the students to be tested with sufficient accuracy.

Correlation between the optimal screening for prediction of referral and outcome yellow flag tool and patient-reported legacy outcome measures in patients undergoing shoulder surgery

BackgroundThe Optimal Screening for Prediction of Referral and Outcome Yellow Flag (OSPRO-YF) Tool is a 10-item multidimensional screening tool utilized to evaluate pain-related psychological traits in individuals with musculoskeletal pain conditions. The validity of postoperatively collected OSPRO-YF is unclear. This study sought to assess validity of the OSPRO-YF by comparing it to PRO scores in both pre- and postoperative settings. HypothesisThe authors hypothesized that OSPRO-YF overall score would correlate with shoulder and global function PROs at pre- and postoperative timepoints. MethodsA review of 101 patients undergoing shoulder surgery by one sports medicine orthopedic surgeon at a large academic institution was conducted. 90 and 54 patients had complete pre- and postoperative PRO responses. OSPRO-YF, American Shoulder and Elbow Surgeons Evaluation Form (ASES), and Patient-Reported Outcomes Measurement Information System Computer Adaptive Test (PROMIS-CAT) were routinely administered before and after surgery at the senior author’s clinic visits. Concurrent validity of OSPRO-YF at either timepoint was assessed by comparing scores with PROs cross-sectionally using Pearson correlations and multiple comparison corrections. ResultsPreoperatively, higher OSPRO-YF total score was associated with greater concurrent PROMIS-CAT Pain Interference (r=0.43; p<0.01) and Depression (r=0.36; p=0.05) and lower ASES (r=-0.34; p<0.01). Higher postoperative OSPRO-YF was also associated with greater concurrent PROMIS-CAT Pain Interference (r=0.43; p<0.01) and Depression (r=0.36; p<0.01) and lower ASES (r=-0.34; p=0.01). ASES had strong correlation with SANE and Pain scores at both pre- and postoperative timepoints. SANE was not significantly associated with OSPRO-YF total score or number of YFs at either timepoints. ConclusionThe study findings support the clinical validity of the 10-item OSPRO-YF tool when administered before or after shoulder surgery. For patients exhibiting suboptimal recovery or those identified as high risk at initial screening, assessment of pain-related psychological distress postoperatively may be particularly beneficial in guiding rehabilitation.

Efficiency of computerized adaptive testing with a cognitively designed item bank.

An item bank is key to applying computerized adaptive testing (CAT). The traditional approach to developing an item bank requires content experts to design each item individually, which is a time-consuming and costly process. The cognitive design system (CDS) approach offers a solution by automating item generation. However, the CDS approach has a specific way of calibrating or predicting item difficulty that affects the measurement efficiency of CAT. A simulation study was conducted to compare the efficiency of CAT using both calibration and prediction models. The results show that, although the predictive model (linear logistic trait model; LLTM) shows a higher root mean square error (RMSE) than the baseline model (Rasch), it requires only a few additional items to achieve comparable RMSE. Importantly, the number of additional items needed decreases as the explanatory rate of the model increases. These results indicate that the slight reduction in measurement efficiency due to prediction item difficulty is acceptable. Moreover, the use of prediction item difficulty can significantly reduce or even eliminate the need for item pretesting, thereby reducing the costs associated with item calibration.

The General Psychopathology 'p' Factor in Adolescence: Multi-Informant Assessment and Computerized Adaptive Testing.

Accumulating evidence supports the presence of a general psychopathology dimension, the p factor ('p'). Despite growing interest in the p factor, questions remain about how p is assessed. Although multi-informant assessment of psychopathology is commonplace in clinical research and practice with children and adolescents, almost no research has taken a multi-informant approach to studying youth p or has examined the degree of concordance between parent and youth reports. Further, estimating p requires assessment of a large number of symptoms, resulting in high reporter burden that may not be feasible in many clinical and research settings. In the present study, we used bifactor multidimensional item response theory models to estimate parent- and adolescent-reported p in a large community sample of youth (11-17 years) and parents (N = 5,060 dyads). We examined agreement between parent and youth p scores and associations with assessor-rated youth global functioning. We also applied computerized adaptive testing (CAT) simulations to parent and youth reports to determine whether adaptive testing substantially alters agreement on p or associations with youth global functioning. Parent-youth agreement on p was moderate (r =.44) and both reports were negatively associated with youth global functioning. Notably, 7 out of 10 of the highest loading items were common across reporters. CAT reduced the average number of items administered by 57%. Agreement between CAT-derived p scores was similar to the full form (r =.40) and CAT scores were negatively correlated with youth functioning. These novel results highlight the promise and potential clinical utility of a multi-informant p factor approach.

Psychometrics of patient-reported outcomes measurement information system in von Willebrand disease, inherited platelet function disorders, and rare bleeding disorders

BackgroundPatient-reported outcomes measurement information system (PROMIS) measures can be used to measure patient-reported outcomes. PROMIS measures, including computer adaptive tests (CATs) and short forms, have demonstrated the ability to adequately assess outcomes in patients with hemophilia. It is, however, unclear if PROMIS measures are suitable for patients with von Willebrand disease (VWD), inherited platelet function disorders (IPFDs), and rare bleeding disorders (RBDs). ObjectivesTo evaluate the feasibility, measurement properties, and relevance of PROMIS measures in adults with VWD, IPFDs, and RBDs. MethodsIn this cross-sectional multicenter study, adults with VWD, IPFDs, and RBDs completed 9 PROMIS measures and the Short Form-36 version 2 (SF-36v2) electronically. Feasibility was determined by the number of completed items and floor/ceiling effects. Measurement properties included construct validity based on a multitrait–multimethod analysis and reliability using the reliability coefficient and greatest lower bound. Relevance was evaluated based on comparison with the Dutch general population. ResultsIn total, 111 patients (median age, 57 years [IQR, 44-67]; 60% VWD, 16% IPFD, 24% RBD) participated. Mean number of items answered varied from 5.3 to 8.7 (range, 4-12) per PROMIS CAT in patients with VWD. Construct validity was supported for all CATs and all instruments had a good reliability (≥0.70). The PROMIS measures had less ceiling effects than the SF-36v2. ConclusionThe PROMIS measures are a feasible, valid, and reliable alternative for the SF-36v2 in patients with primarily nonsevere forms of VWD. The relevance of the selected measures was limited. Additional research is necessary to evaluate the PROMIS measures in adults with IPFDs and RBDs.

Robustness of Computer Adaptive Tests to the Presence of Item Preknowledge: A Simulation Study

Item preknowledge describes a scenario where candidates may have access to the some of the test items prior to the test administration. This involves the sharing of test materials and/or answers and it is difficult to identify the individuals with item preknowledge or the shared materials of test. Nevertheless, it is essential to investigate the ‘item preknowledge’ problem because it can significantly affect the validity of test results. It is believed that traditional linear tests are more robust to this type of aberrant response behavior than adaptive tests. In this context, the aim of this study is to examine the effect of item preknowledge on computer adaptive tests and to try to identify the conditions under which adaptive tests are most resistant to the item preknowledge. With this purpose, a Monte Carlo simulation study was performed and 28 different conditions were examined. The results of the study indicated the EAP estimation method provided better measurement precision than ML over all conditions. When 2PL and 3PL IRT models were compared, it was observed that 2PL had higher precision at most of the conditions. However, when the aberrancy ratio increased and reached to the 20% for both individuals and items, 3PL outperformed the 2PL model and gave the best results with the EAP combination. The results were discussed in line with literature on item preknowledge and CAT and implications for practitioners and further research were provided.

Age-normed patient-reported outcome measures among cancer survivors.

11122 Background: Patient-reported outcome measures (PROMs) are instruments used to collect health-related outcomes that are derived completely from the patient without interpretation from healthcare providers. Monitoring and subsequent intervention based on PROMs collection has demonstrated efficacy in clinical trial settings. This has resulted in implementation of PROMs in routine clinical cancer practices throughout the nation. PROMs have been normed to the general population, but establishing the norms among cancer patients is essential to understanding how PROMs can be evaluated on an individual basis in routine clinical cancer care. This study aims to characterize age-normed PROMs scores among survivors of cancer using the National Institute of Health’s Patient-Reported Outcome Measures Information System (PROMIS) depression, fatigue, pain interference, and physical function domains. Methods: Routine collection of PROMs using 4 domains (depression, fatigue, pain interference, and physical function) of PROMIS were offered to all patients with a diagnosis of cancer of any disease site at visits with an oncologic provider using computer adaptive testing. All patients ≥18 years old with a completed PROM at least 2 years since their diagnosis of cancer were eligible for inclusion in the analysis. Only completed PROMs were included in the final analysis. Generalized estimating equation models were used to assess the relationship between age and the estimated mean T-score for each PROMs domain, considering repeat measures within a single patient. Results: A total of 3,636 patients were included in this retrospective cohort study with a total of 26,173 completed PROMs among all 4 domains. Mean age at diagnosis was 61.2 years (SD=12.44), 64% (n=2324) were female, 68% (n=2,461) identified as White, and 25% (n=893) identified as Black. For fatigue, mean T-score ranged from 48.4 points (SD=9.6) among 18-29 years olds to 56.5 points (SD=10.1) among 90-99 years olds, with no significant change with age (p=0.27). For depression, mean T-score ranged from 48.9 points (SD=9.0) among 60-69 year olds to 51.1 points (SD=8.8) among 80-89 year olds with a 0.3 point/decade decrease in T-score (p=0.01). Pain interference T-scores ranged from 48.6 points (SD=10.5) among 18-29 year olds to 55.0 points (SD=9.4) among 80-89 year olds with a 0.4 point/decade average increase (p&lt;0.001). The largest differences were observed in physical function, where scores ranged from 53.5 points (SD=11.0) among 18-29 year olds to 34.3 points (SD=9.2) among 90-99 year olds. There was a 2.1 point/decade decrease in T-score (p&lt;0.001). Conclusions: Among survivors of cancer, mean PROMIS scores differed by age in the depression, pain interference, and physical function domains but not fatigue. These age-normed PROMIS T-scores, which are required to understand individualized assessments of PROMs, are essential as PROMs are integrated into routine cancer care.

Patient Reported Outcome Measurement Information System (PROMIS) in routine oncological care: A literature review and survey of users.

e23192 Background: Patient-reported outcome measures (PROMs) are critical for patient-centered care in oncology, as they allow patients to communicate with their care team about a range of symptoms using well-validated tools. However, successful use of PROMs in routine care depends on the clinical user experience: the opinions and experiences of the clinicians and staff who administer and make treatment decisions using PROMs. This study characterizes the use of the NIH-sponsored Patient Reported Outcomes Measurement Information System (PROMIS) in routine oncological care through a literature review and user survey. Methods: First, we conducted a narrative review of papers published from 2010 to June 2023 to identify breadth of PROMIS use in routine care. We screened 262 abstracts and identified 10 articles describing routine care in patients with cancer. We coded for institution type, PROM administration methods, and types of PROMIS measures. Next, we conducted a survey to solicit the opinions and experiences of diverse stakeholders implementing PROMIS in routine care. We contacted 209 known users of PROMIS (published authors, PROMIS API clients, and respondent referrals). Item frequencies were used to characterize survey responses. Chi-square tests were used to test differences between users in oncology and other fields. Results: Review. Nine of the 10 institutions using PROMIS in routine cancer care were academic medical centers. Articles reported on data from 2011-2021. The most popular PROMIS measure was PROMIS Global Health which provides summary scores of physical and mental health. Other popular domains included fatigue (4), physical function (3), depression (3), and anxiety (3), all measured by PROMIS computerized adaptive tests (CATs). They were most often administered in the clinic waiting room (7) or online before appointments (6). Most institutions (6) had PROMIS results automatically integrated into the electronic health record. Survey. We had 89 respondents (6 from oncology). PROMIS users in oncology were more likely to have structured training on the use of PROMIS than other users (χ2 (1,79) = 6.0, p = 0.01) and less likely to give patients direct access to their results (χ2 (1,72) = 6.0, p = 0.01). Users in oncology did not differ from the whole sample in opinions of PROMIS (% agree): users viewed PROMIS as worthwhile (86%), clinically relevant (85%), easy to administer (76%), statistically reliable (73%), and beneficial to treatment monitoring (68%), but were less confident that PROMIS improves doctor-patient communication (57%), treatment decision-making (54%), or patient satisfaction (35%). Conclusions: Conclusions PROMIS has been successfully integrated into routine oncological care in several academic medical centers. While adoption is not widespread, these initial results suggest that PROMIS is a useful tool for symptom monitoring in cancer care.

Validation of patient reported outcomes measurement information system anxiety item bank computer adaptive testing in solid organ transplant recipients

Validation of patient reported outcomes measurement information system anxiety item bank computer adaptive testing in solid organ transplant recipients

Meta-Analysis of the Effects of Computerized Adaptive Testing on the Motivation and Emotion of Examinees

Abstract: Findings on the motivational and emotional effects of computerized adaptive testing (CAT) compared to nonadaptive testing (NCAT) have been heterogeneous and inconclusive so far. This meta-analysis aims to determine the specific effects of CAT better by analyzing several moderator variables related to the test, the examinee, and the test setting. To be included in the meta-analysis, studies needed to be original, correlational, quasi-experimental, or experimental studies in English, German, or Chinese, and they needed to compare CAT to NCAT regarding motivational and/or emotional states. A three-level random-effects model with motivation, negative emotion, and positive emotion as dependent variables was estimated. The risk of bias and publication bias were examined. Twenty-seven studies with a total of 190 effect sizes were analyzed. The overall effect of CAT was not significant for motivation and negative emotion and significant for positive emotion after publication bias correction. The moderator analysis showed significant moderator effects for “test speededness” on motivation and positive emotion, “instantaneous feedback on given responses” and “possibility of response revision” on positive emotion, and “average difficulty of presented items” on negative emotion. For CAT configurations, it is recommended to avoid speededness, present easy items, allow for response revision, and provide feedback.

TRANSFIGURATIONS OF THE COMPUTERIZED MATHEMATICS FINAL EXAM: HOW WAS THE RESULTS OF COMPUTERIZED ADAPTIVE TEST-BASED ASSESSMENT?

Abstract Computerized Adaptive Test (CAT) allows the use of targeted tests, that is, each test taker obtains items that match his or her ability level. The purpose of this research is to find out the final semester exam with the CAT model. The research methodology is descriptive quantitative. Purposive sampling technique was used to take a sample of 73 students. The results of this study indicate that the CAT used in this semester final exam based on Moodle can select test items given to test takers adaptively according to the user's ability level. The very high ability category was 42 students, the high ability category was 11 students, the medium ability category was 9 students, the low ability category was 4 students, and the very low ability category was 7 students. Overall, the ability of learners is in the high category.

Linking QuickDASH and PROMIS Upper-Extremity Computer-Adaptive Test Scores in Hand Surgery: A Crosswalk Study

Assessment of patient-reported outcome measures (PROMs) for hand and upper-extremity surgery patients using measures such as the Quick Disabilities of the Arm, Shoulder, and Hand (qDASH), as well as general measures including the Patient-Reported Outcomes Measurement Information System Upper Extremity Physical Function domain via a Computer-Adaptive Test (PROMIS UE CAT), has become commonplace. The aim of this study was to link, for crosswalking, the qDASH measure to both versions of the PROMIS UE CAT (v1.2 and v2.0). We included 18,944 hand and upper-extremity patients who completed both versions of the PROMIS UE CAT and the qDASH at the same clinical encounter. Shoulder pathology was excluded. Score linkage was performed using the R package equate, and multiple equating models (linear regression, identity, mean, linear, equipercentile, and circle-arc models) were used to establish crosswalk tables. Mean qDASH and PROMIS UE CAT v1.2 scores were 38.2 (SD= 23.1) and 36.6 (SD= 9.8), respectively. Mean qDASH and PROMIS UE CAT v2.0 scores were 37.3 (SD= 21.8) and 38.3 (SD= 10.4), respectively. Pearson correlations had very strong linear relationships between the qDASH and the PROMIS UE CAT v1.2 and PROMIS UE CAT v2.0 (r=-0.83 [-0.84,-0.92] and r=-0.80 [-0.81,-0.80], respectively). For the equipercentile equating models, the intraclass correlation coefficient (ICC) had very strong positive relationships to linking measures with ICC= 0.85 (0.84, 0.86) for the qDASH-UE CAT v1.2 crosswalk and ICC= 0.83 (0.82, 0.84) for the qDASH-UE CAT v2.0 crosswalk. The linkages establish crosswalk tables using equipercentile equating models to convert the PROMIS UE CAT v1.2 and v2.0 scores to the qDASH and vice versa. This study provides crosswalk tables for commonly collected PROMs in hand surgery, increasing the comparability of results between centers using different PROMs to study the same conditions or treatments.

Model-Agnostic Adaptive Testing for Intelligent Education Systems via Meta-learned Gradient Embeddings

The field of education has undergone a significant revolution with the advent of intelligent systems and technology, which aim to personalize the learning experience, catering to the unique needs and abilities of individual learners. In this pursuit, a fundamental challenge is designing proper test for assessing the students’ cognitive status on knowledge and skills accurately and efficiently. One promising approach, referred to as Computerized Adaptive Testing (CAT), is to administrate computer-automated tests that alternately select the next item for each examinee and estimate their cognitive states given their responses to the selected items. Nevertheless, existing CAT systems suffer from inflexibility in item selection and ineffectiveness in cognitive state estimation, respectively. In this paper, we propose a Model-Agnostic adaptive testing framework via Meta-leaned Gradient Embeddings, MAMGE for short, improving both item selection and cognitive state estimation simultaneously. For item selection, we design a Gradient Embedding based Item Selector (GEIS) which incorporates the concept of gradient embeddings to represent items and selects the best ones that are both informative and representative. For cognitive state estimation, we propose a Meta-learned Cognitive State Estimator (MCSE) to automatically control the estimation process by learning to learn a proper initialization and dynamically inferred updates. Both MCSE and GEIS are inherently model-agnostic, and the two modules have an ingenious connection via meta-learned gradient embeddings. Finally, extensive experiments evaluate the effectiveness and flexibility of MAMGE.

Psychometric Assessment of an Item Bank for Adaptive Testing on Patient-Reported Experience of Care Environment for Severe Mental Illness: Validation Study.

The care environment significantly influences the experiences of patients with severe mental illness and the quality of their care. While a welcoming and stimulating environment enhances patient satisfaction and health outcomes, psychiatric facilities often prioritize staff workflow over patient needs. Addressing these challenges is crucial to improving patient experiences and outcomes in mental health care. This study is part of the Patient-Reported Experience Measure for Improving Quality of Care in Mental Health (PREMIUM) project and aims to establish an item bank (PREMIUM-CE) and to develop computerized adaptive tests (CATs) to measure the experience of the care environment of adult patients with schizophrenia, bipolar disorder, or major depressive disorder. We performed psychometric analyses including assessments of item response theory (IRT) model assumptions, IRT model fit, differential item functioning (DIF), item bank validity, and CAT simulations. In this multicenter cross-sectional study, 498 patients were recruited from outpatient and inpatient settings. The final PREMIUM-CE 13-item bank was sufficiently unidimensional (root mean square error of approximation=0.082, 95% CI 0.067-0.097; comparative fit index=0.974; Tucker-Lewis index=0.968) and showed an adequate fit to the IRT model (infit mean square statistic ranging between 0.7 and 1.0). DIF analysis revealed no item biases according to gender, health care settings, diagnosis, or mode of study participation. PREMIUM-CE scores correlated strongly with satisfaction measures (r=0.69-0.78; P<.001) and weakly with quality-of-life measures (r=0.11-0.21; P<.001). CAT simulations showed a strong correlation (r=0.98) between CAT scores and those of the full item bank, and around 79.5% (396/498) of the participants obtained a reliable score with the administration of an average of 7 items. The PREMIUM-CE item bank and its CAT version have shown excellent psychometric properties, making them reliable measures for evaluating the patient experience of the care environment among adults with severe mental illness in both outpatient and inpatient settings. These measures are a valuable addition to the existing landscape of patient experience assessment, capturing what truly matters to patients and enhancing the understanding of their care experiences. ClinicalTrials.gov NCT02491866; https://clinicaltrials.gov/study/NCT02491866.