Difficulty Of Individual Items Research Articles

Education Research: Quality and Validity Evidence for a National In-Training Examination for Epilepsy Fellows.

Epilepsy education has been transformed over the past 2 decades, leading to a need for structured formative assessment tools. The American Epilepsy Society developed the Epilepsy Fellowship In-Training Examination (EpiFITE) to provide high-quality formative assessment for fellows, to stimulate program improvement, and to guide future learning and teaching. The aim of this study was to explore validity evidence for the EpiFITE in meeting these goals. Validity evidence was sought from multiple sources. The content of the examination was linked to the American Board of Psychiatry and Neurology blueprint for initial certification in epilepsy, and items were developed by trained experts. Internal structure was studied using internal consistency and item analysis. Surveys of fellows and fellowship directors focused on the examination experience (response process) and how results influenced fellow assessment, future learning, and program improvement (relationship to other variables and consequences). The EpiFITE was first administered in 2020, with 172 examinees from 67 programs. By 2022 (year 3), the EpiFITE was completed by 195 epilepsy fellows from 77 programs. The overall mean score of the examination was stable from year to year, and the committee predicted the difficulty of individual items with a high degree of accuracy. The examination had high internal consistency (Cronbach α 0.76-0.81). The median item discrimination index ranged from 0.17 in 2020 to 0.21 in 2022. Discrimination indices were lower (mean ≤0.10) for items that were either very easy or very difficult and significantly higher (mean >0.20) for other items. Program directors and epilepsy fellows agreed the examination questions were appropriate and agreed that the EpiFITE helped them identify areas for self-directed learning. Program directors also found the examination helpful in identifying areas of strength and areas for improvement within their programs. There are several sources of evidence of the quality and validity of the EpiFITE. By exploring this validity evidence, we have identified several best practices in the development and evaluation of a subspecialty examination, and this experience could be helpful for developers of in-training examinations in other subspecialties.

Cochlear Implant Quality of Life (CIQOL): Development of a Profile Instrument (CIQOL-35 Profile) and a Global Measure (CIQOL-10 Global).

Purpose Valid, reliable, and efficient patient-reported outcome measures are needed to quantify quality of life (QOL) outcomes after cochlear implantation to supplement information obtained from performance-based outcomes. We previously developed the Cochlear Implant Quality of Life (CIQOL) item bank to serve as the source of items for subsequent instruments. This study reports the development and psychometric properties for 2 of these new instruments, the CIQOL-35 Profile and the CIQOL-10 Global. Method Cochlear implant (CI) users referred from the CIQOL Development Consortium (n = 371), consisting of 20 CI centers across the United States, provided responses to the 81-item CIQOL item bank, which are grouped into 6 QOL domains (communication, emotional, entertainment, environment, listening effort, and social). Responses to the 81 CIQOL items were analyzed using item response theory to determine individual item difficulty, discrimination, and model fit to select the set of items for the profile instrument and global measure that would optimize their measurement characteristics. Results The 35-item CIQOL-35 Profile instrument assesses outcomes represented in the 6 domains of the CIQOL final item pool. The 10-item CIQOL-10 Global measure produces a single, overall QOL score. After ensuring the upper and lower ends of the item difficulty continuum were represented (item difficulty range: -2.48 to 2.47), the items with the highest discrimination ability for each domain were selected for the CIQOL-35 Profile instrument (discrimination range: 0.67-1.37). Items were selected for the CIQOL-10 Global measure in a similar manner. Conclusion The CIQOL-35 Profile and CIQOL-10 Global instruments provide psychometrically sound and efficient measures that can be used to assess QOL in adult CI users in both clinical and research settings. Supplemental Material https://doi.org/10.23641/asha.9745010.

Intelligibility and comprehensibility in English as a lingua franca: nativized English in Japanese

The present study examines international intelligibility and comprehensibility of nativized English lexis from Japanese as determined by tertiary student listeners in four countries. After an overview of nativized English in Japanese, form and attitudes, the study queries which linguistic features of Japanized English lexis and short expressions reduce intelligibility and comprehensibility for listeners from Indonesia, the Philippines, South Korea, and the United States. The results indicate that morphological and semantic modifications tend to be more challenging to the Indonesian, Filipino, and US listener groups than phonological ones, while South Korean listeners had an advantage with nativized expressions. Rank orders of the difficulty of individual items tended to be shared by the Filipino and US groups. Listener factors were identified including first language (L1) influence, English proficiency, and sociocultural background. Further considered are linguistic complexity, cultural, and historical linguistic flows, Englishes as dialectal variation, and ELF pronunciation pedagogy.

Setting standards in knowledge assessments: Comparing Ebel and Cohen via Rasch

Introduction: It is known that test-centered methods for setting standards in knowledge tests (e.g. Angoff or Ebel) are problematic, with expert judges not able to consistently predict the difficulty of individual items. A different approach is the Cohen method, which benchmarks the difficulty of the test based on the performance of the top candidates.Methods: This paper investigates the extent to which Ebel (and also Cohen) produces a consistent standard in a knowledge test when comparing between adjacent cohorts. The two tests are linked using common anchor items and Rasch analysis to put all items and all candidates on the same scale.Results: The two tests are of a similar standard, but the two cohorts are different in their average abilities. The Ebel method is entirely consistent across the two years, but the Cohen method looks less so, whilst the Rasch equating itself has complications – for example, with evidence of overall misfit to the Rasch model and change in difficulty for some anchor items.Conclusion: Based on our findings, we advocate a pluralistic and pragmatic approach to standard setting in such contexts, and recommend the use of multiple sources of information to inform the decision about the correct standard.

Assessment of planning ability: Psychometric analyses on the unidimensionality and construct validity of the Tower of London Task (TOL-F).

The Tower of London (TOL) is widely used to assess planning ability as a prototypical executive function in healthy and clinical populations. Despite its popularity, there is still no consensus on (a) whether the TOL measures a psychometrically unidimensional trait, and (b) how differences in problem structure relate to the difficulty of individual items. Employing the framework of factor analysis and item response theory we investigated these issues of construct validity in a sample of 798 participants (443 female; 16 to 84 years). Participants worked on the TOL-Freiburg version (TOL-F) comprising a set of 24 4- to 6-move problems, which--based on comprehensive cognitive task analyses-systematically differ with regard to several structural problem parameters (minimum number of moves, search depth, goal hierarchy). Results revealed that TOL-F performance is mainly explained by 1 major factor, while further minor factors additionally account for smaller, but possibly still informative, shares of variance. Individual item difficulties can be predicted by the experimentally varied problem parameters to a high degree (r = .89) and can be considered stable across different levels of age, sex, education, and planning ability. These findings demonstrate the TOL-F's construct validity as measuring planning ability in terms of an essentially unidimensional cognitive function while adhering to theory-driven concepts of task difficulty. TOL-F task performance hence represents an accurate, robust, and theoretically grounded estimation of a participant's planning ability. The results further highlight the merit of using established concepts from experimental psychology for improving neuropsychological assessment.

A Comparison of Two Low-Stakes Methods for Administering a Program-Level Biology Concept Assessment.

Concept assessments are used commonly in undergraduate science courses to assess student learning and diagnose areas of student difficulty. While most concept assessments align with the content of individual courses or course topics, some concept assessments have been developed for use at the programmatic level to gauge student progress and achievement over a series of courses or an entire major. The broad scope of a program-level assessment, which exceeds the content of any single course, creates several test administration issues, including finding a suitable time for students to take the assessment and adequately incentivizing student participation. These logistical considerations must also be weighed against test security and the ability of students to use unauthorized resources that could compromise test validity. To understand how potential administration methods affect student outcomes, we administered the Molecular Biology Capstone Assessment (MBCA) to three pairs of matched upper-division courses in two ways: an online assessment taken by students outside of class and a paper-based assessment taken during class. We found that overall test scores were not significantly different and that individual item difficulties were highly correlated between these two administration methods. However, in-class administration resulted in reduced completion rates of items at the end of the assessment. Taken together, these results suggest that an online, outside-of-class administration produces scores that are comparable to a paper-based, in-class format and has the added advantages that instructors do not have to dedicate class time and students are more likely to complete the entire assessment.

Measuring meta-ignorance through the lens of confidence: examining students' redox misconceptions about oxidation numbers, charge, and electron transfer

This manuscript describes the relationship between students' redox understandings and confidence as measured by the Redox Concept Inventory (ROXCI) which assesses symbolic and particulate redox concepts. The ROXCI was administered to two samples of 1st- and 2nd-semester general chemistry students after the students were taught and tested on redox concepts in their classrooms. Cluster analysis was used to identify groups of students with similar response patterns, based upon both total scores and average confidence on the ROXCI. Three clusters of students were identified in both samples: students with (1) moderate total scores and high confidence, (2) low total scores and low confidence, and (3) low total scores but high confidence. Clusters were further analyzed at an individual item level using average confidence, individual item difficulties, and the Confidence Discrimination Quotient (CDQ). Findings align with the Dunning–Kruger effect,i.e.in which students demonstrated a false sense of confidence regarding their own poor performance, and therefore, exemplify meta-ignorance. Descriptions of the clusters, example misconceptions held by the students regarding oxidation numbers and electron transfer, and the implications of this research are discussed.

A Classical Test Theory Analysis of the Light and Spectroscopy Concept Inventory National Study Data Set

This paper is the first in a series of investigations into the data from the recent national study using the Light and Spectroscopy Concept Inventory (LSCI). In this paper, we use classical test theory to form a framework of results that will be used to evaluate individual item difficulties, item discriminations, and the overall reliability of the LSCI. We perform an analysis of individual students’ normalized gains, providing further insight into the prior results from this data set. This investigation allows us to better understand the efficacy of measuring student achievement using the LSCI. Future papers will discuss our investigation of the data from the recent national study using item response theory (IRT).

Comparison of student performance on “selected‐response” versus “constructed‐response” question formats in a medical neuroscience laboratory practical examination

Laboratory practical exams are a component of knowledge-based assessment for first-year medical students in our Patient-Centered Learning (PCL) curriculum and have historically been constructed in a selected response (multiple choice) format. This study was conducted in order to address influences that question stimulus and/or response formats may have on assessment of student learning in a teaching laboratory setting. A neuroscience laboratory practical exam, previously included as an actual part of PCL assessment, was offered as a pre-exam, self-assessment exercise to two classes of students who were completing the Year 1 neuroscience curriculum. Randomized groups of students received the exercise in either selected-response or constructed-response (fill-in-the-blank) formats and student performance results were compared. Students performed significantly better on the exercise constructed in selected-response format. Comparisons of individual item difficulty and discrimination indices indicated that while asking a student to reflect on multiple options and come to an informed decision about the answer they choose is valid and reliable, “cueing” based on recognition and recall rather than depth of knowledge and understanding can explain student performance differences, raising legitimate issues for continuous improvement in curriculum and assessment. Supported by UNDSMHS.

Evaluating existing measures of health numeracy using item response theory

Objective To evaluate existing measures of health numeracy using item response theory (IRT). Methods A cross-sectional study was conducted. Participants completed assessments of health numeracy measures including the Lipkus expanded health numeracy scale (Lipkus), and the Medical Data Interpretation Test (MDIT). The Lipkus and MDIT were scaled with IRT utilizing the two-parameter logistic model. Results Three-hundred and fifty-nine (359) participants were surveyed. Classical test theory parameters and IRT scaling parameters of the numeracy measures found most items to be at least moderately discriminating. Modified versions of the Lipkus and MDIT were scaled after eliminating items with low discrimination, high difficulty parameters, and poor model fit. The modified versions demonstrated a good range of discrimination and difficulty as indicated by the test information functions. Conclusion An IRT analysis of the Lipkus and MDIT indicate that both health numeracy scales discriminate well across a range of ability. Practice implications Health numeracy skills are needed in order for patients to successfully participate in their medical care. The accurate assessment of health numeracy may help health care providers to tailor patient education interventions to the patient's level of understanding and ability. Item response theory scaling methods can be used to evaluate the discrimination and difficulty of individual items as well as the overall assessment.

Impoverished cue support enhances subsequent retention: Support for the elaborative retrieval explanation of the testing effect

In three experiments, we investigated the role of transfer-appropriate processing and elaborative processing in the testing effect In Experiment 1, we examined whether the magnitude of the testing effect reflects the match between intervening and final tests by factorially manipulating the type of intervening and final tests. Retention was not enhanced for matching, relative to mismatching, intervening and final tests, contrary to the transfer-appropriate-processing view. In Experiment 2, we examined final retention as a function of the number of cues needed to retrieve items on intervening cued recall tests. In this case, fewer retrieval cues were associated with better memory on the final test. Experiment 3 replicated the findings of Experiment 2 while controlling for individual item difficulty and directly manipulating the number of cues present. These findings suggest that an intervening test may be most beneficial to final retention when it provides more potential for elaborative processing

The potential for experimenter bias effects in word recognition experiments.

Word recognition experiments that involve comparisons between two different matched sets of words allow for possible experimenter bias if there are many equally well-matched pairs to choose from. The possible extent of this bias depends on the experimenter's intuitive knowledge of the likely difficulty of individual items. This is assessed by asking a number of experimenters to decide which of two frequency-matched words would produce the fastest reaction time in a lexical decision experiment, and then comparing their predictions with actual data. All experimenters demonstrated substantial above-chance accuracy, which was unrelated to the amount of experience they had in the field. It was concluded that the experimenters could potentially produce spurious effect sizes ranging from 16 to 38 msec.

The Development and Validation of a Reliable Alternate Form for Raven's Standard Progressive Matrices

The development of an alternate form for Raven's Standard Progressive Matrices (SPM) Test is described. Items for each of the original 60 items of the SPM were developed to be comparable to the corresponding original items in terms of the underlying strategy and difficulty. An alternate form reliability analysis on a diverse group of 449 children (77 African-American, 122 Asian, 54 Filipino, 156 Latino/Hispanic, 38 white, and 2 other) showed an alternate form reliability coefficient of .90. Kuder-Richardson reliabilities of the newly developed alternate and the SPM were identical at .94: A Kolmogorov-Smirnov two-sample test, moreover, revealed no significant differences in central tendency, dispersion, and skewness for distributions of individual item difficulties. In addition, the two tests showed comparable predictive validity coefficients. The alternate form resolves one limitation of the SPM and could provide widespread utility as a research tool.

On Using the Shewhart Control Chart To Display Item Analysis Data

A procedure was described by Rogers (1978) for using statistical control charts to display test item analysis data. Rogers' charts in volved plotting individual item difficulty indices with control limits and centerline values derived from an implicitly assumed mean item difficulty of 50%. The author of the present paper contends that a more useful representation of test item characteristics can be ob tained by adopting the usual control charting convention of allow ing the data itself to determine limit and centerline values. Then these values can be compared to any criterion difficulty level(s) deemed appropriate for a given set of test items. Although control charts cannot always be applied to item analysis, the author de scribes two specific situations where they can be used.

A Validation Study of a Transferred Group Intelligence Test

Abstract The purpose of the research was to conduct a validation study of a transferred group intelligence test. The Lorge-Thorndike Intelligence Test, Level A, Form 1 was translated to Hebrew and administered to an Israeli sample comparable to the original norming group. The results obtained were concerned with variability, reliability, validity, item analysis, intercorrelations among subtests, and factor analysis. While few comparisons of the Israeli sample with the original data were made, the validation of the transferred test was examined in the light of other criteria for evaluating a new test. The suitability of the transferred test for Israeli high achievers from high-status families was supported by the results. The major conclusions were: (1) the total mean IQ scores of Israeli students were approximately the same as the American mean; (2) mean raw scores increased with age; (3) relative difficulty of individual items remained stable through translation and administration in Israeli culture, in ...

Effects of directions regarding guessing on item statistics of a multiple-choice vocabulary test.

A 100-item vocabulary test containing 80 words of normal difficulty, ten extremely difficult words, and ten nonexistent “words” was administered to a group of 801 Annapolis first-year students. There were three sets of directions, one third of the population being told nothing about guessing, one third being warned against guessing, and one third being encouraged to guess. Directions not to guess were found to have the most direct effect on the number of questions attempted; students who were told nothing about guessing acted very similarly to students who were encouraged to guess. The tendency to guess proved to have an almost negligible positive relation to ability in the area of the test. The reliability of the test was lowered by the presence of the special items. Individual item difficulties were affected by the directions, deltas being higher where there was least guessing. Biserial correlations of the ordinary items were adversely affected, but to a slight extent only, by the presence of the special items. The very difficult items were found to react in exactly the same way as the nonsense items.

EFFECTS OF DIRECTIONS REGARDING GUESSING ON ITEM STATISTICS OF A MULTIPLE‐CHOICE VOCABULARY TEST

ABSTRACTA 100‐item vocabulary test containing 80 words of normal difficulty, ten extremely difficult words, and ten nonexistent “words” was administered to a group of 801 Annapolis first‐year students. There were three sets of directions, one third of the population being told nothing about guessing, one third being warned against guessing, and one third being encouraged to guess.Directions not to guess were found to have the most direct effect on the number of questions attempted; students who were told nothing about guessing acted very similarly to students who were encouraged to guess. The tendency to guess proved to have an almost negligible positive relation to ability in the area of the test. The reliability of the test was lowered by the presence of the special items. Individual item difficulties were affected by the directions, deltas being higher where there was least guessing. Biserial correlations of the ordinary items were adversely affected, but to a slight extent only, by the presence of the special items. The very difficult items were found to react in exactly the same way as the nonsense items.