Differences In Item Difficulty Research Articles

Large Language Models in Medical Education: Comparing ChatGPT- to Human-Generated Exam Questions.

Creating medical exam questions is time consuming, but well-written questions can be used for test-enhanced learning, which has been shown to have a positive effect on student learning. The automated generation of high-quality questions using large language models (LLMs), such as ChatGPT, would therefore be desirable. However, there are no current studies that compare students' performance on LLM-generated questions to questions developed by humans. The authors compared student performance on questions generated by ChatGPT (LLM questions) with questions created by medical educators (human questions). Two sets of 25 multiple-choice questions (MCQs) were created, each with 5 answer options, 1 of which was correct. The first set of questions was written by an experienced medical educator, and the second set was created by ChatGPT 3.5 after the authors identified learning objectives and extracted some specifications from the human questions. Students answered all questions in random order in a formative paper-and-pencil test that was offered leading up to the final summative neurophysiology exam (summer 2023). For each question, students also indicated whether they thought it had been written by a human or ChatGPT. The final data set consisted of 161 participants and 46 MCQs (25 human and 21 LLM questions). There was no statistically significant difference in item difficulty between the 2 question sets, but discriminatory power was statistically significantly higher in human than LLM questions (mean = .36, standard deviation [SD] = .09 vs mean = .24, SD = .14; P = .001). On average, students identified 57% of question sources (human or LLM) correctly. Future research should replicate the study procedure in other contexts (e.g., other medical subjects, semesters, countries, and languages). In addition, the question of whether LLMs are suitable for generating different question types, such as key feature questions, should be investigated.

Fourteen-item perceived stress scale assessment using item response theory among pregnant women.

The current study aimed to assess the psychometric properties of the Spanish language version of the 14-item Perceived Stress Scale (PSS-S) in a population of pregnant women who speak Spanish in Peru using item response theory (IRT). Our study consisted of 5,435 pregnant women who participated in the Pregnancy Outcomes Maternal and Infant Study (PrOMIS) cohort in Peru. Exploratory and confirmatory factor analyses were conducted to determine dimensionality of the scale in this population, and item response theory was conducted to determine the applicability of the PSS. The PSS consisted of a 2-factor questionnaire measuring perceived stress and coping capacity accounting for 77% of variability. The IRT analysis showed differences in item difficulty and discrimination. Item difficulty represents the level of the latent construct where 50% of respondents endorse a particular response, and item discrimination determines the rate of change of the probability of endorsing an item for differing ability levels. For the first factor, perceived stress, item 12 was the least difficult and item 2 was the most difficult. For the second factor, coping capacity, item 9 was the least difficult and item 6 was the most difficult. The Spanish version of the 14-item PSS can be a useful assessment tool for perceived stress, but more IRT should be done to delve further into the psychometric properties of the questionnaire to inform clinicians and policy makers more appropriately.

Why are certain items more difficult than others in a competency test for music–related argumentation?

This paper examines why certain items in a competency test for music-related argumentation are more difficult than others. Based on previous studies on school-related achievement tests, the authors assume that differences in item difficulty are related to different item characteristics or combinations of characteristics. In this study, the item characteristics of a test for music-related argumentation were first identified and coded. Three domains were identified as contributing to item difficulty: cognitive requirements, knowledge, and formal item features. Second, multiple linear regression analyses were conducted with the item characteristics as predictors of item difficulty, which had been estimated in a prior study. A comparison of three regression models confirmed that the model holding four predictors of the domain “cognitive requirements” best fit the study dataRadj2=0.71. The strongest predictor in the final model was “reference to musical attributes” β=0.46,β=0.51, followed by “cross-sentence argumentation” β=0.37 and “dialogical argumentation” β=0.20. These results indicate that the difficulty of an item increased most when participants had to refer to musical attributes to solve the task. The items that required the participants to provide cross-sentence or dialogical argumentation were more challenging as well. The findings regarding the relations between item characteristics and item difficulty contribute to a better understanding of music-related argumentative competence, with important implications for the music classroom.

Flynn effects are biased by differential item functioning over time: A test using overlapping items in Wechsler scales

The items of intelligence tests can demonstrate differential item functioning across different groups: cross-sample differences in item difficulty or discrimination, independently of any difference of ability. This is also true of comparisons over time: as the cultural context changes, items may increase or decrease in difficulty. This phenomenon is well-known, but its impact on estimates of the Flynn effect has not been systematically investigated. In the current study, we tested differential item functioning in a subset of 111 items common to consecutive versions of the French WAIS-R (1989), WAIS-III (1999) and/or WAIS-IV (2009), using the three normative samples (total N = 2979). Over half the items had significant differential functioning over time, generally becoming more difficult from one version to the next for the same level of ability. The magnitude of differential item functioning tended to be small for each item separately, but the cumulative effect over all items led to underestimating the Flynn effect by about 3 IQ points per decade, a bias close to the expected size of the effect itself. In this case, this bias substantially affected the conclusions, even creating an ersatz negative Flynn effect for the 1999–2009 period, when in fact ability increased (1989–1999) or stagnated (1999–2009) when accounting for differential item functioning. We recommend that studies of the Flynn effect systematically investigate the possibility of differential item functioning to obtain unbiased ability estimates.

Challenging to Read, Easy to Comprehend? Effects of Linguistic Demands on Secondary Students’ Text Comprehension in Physics

Students with high reading proficiency typically achieve better results in science assessments, indicating the importance of reading proficiency. Since the process of reading is a complex interaction between properties of a text and a reader, the linguistic demands of a text might affect text comprehension. Certain linguistic features, such as complex syntactic structures and low word frequency, have been found to create higher cognitive load. However, studies investigating the influence of linguistic features on test item difficulty and students’ text comprehension in science have hitherto let to inconclusive results. The present study investigates whether the linguistic demands of expository text affect German students’ text comprehension in the domain of physics. Within an experimental study, we presented three introductory texts on different subtopics of thermodynamics and 27 single-select, multiple-choice items to 812 secondary school students (grades 7–9). Items measured students’ text comprehension (dependent variable); the linguistic demands of each text were systematically varied across three levels (independent variables) while other features of text quality and content were held constant. The results of the item response theory analysis indicated no consistent differences in item difficulty across levels of linguistic demands. Moreover, differential analyses of subgroups presented no consistent differences in solution frequencies of items related to different linguistic demands. Furthermore, while the highest linguistic demand of the texts led the students to perceive a lower comprehensibility, their text comprehension was not affected. Hence, this study provides evidence that the influence of linguistic features on text comprehension is at most low and might be overestimated in present discussions.

Assessment methods for determining small changes in hearing performance over time.

Although the behavioral pure-tone threshold audiogram is considered the gold standard for quantifying hearing loss, assessment of speech understanding, especially in noise, is more relevant to quality of life but is only partly related to the audiogram. Metrics of speech understanding in noise are therefore an attractive target for assessing hearing over time. However, speech-in-noise assessments have more potential sources of variability than pure-tone threshold measures, making it a challenge to obtain results reliable enough to detect small changes in performance. This review examines the benefits and limitations of speech-understanding metrics and their application to longitudinal hearing assessment, and identifies potential sources of variability, including learning effects, differences in item difficulty, and between- and within-individual variations in effort and motivation. We conclude by recommending the integration of non-speech auditory tests, which provide information about aspects of auditory health that have reduced variability and fewer central influences than speech tests, in parallel with the traditional audiogram and speech-based assessments.

Swiss general internal medicine board examination: quantitative effects of publicly available and unavailable questions on question difficulty and test performance.

Formerly, a substantial number of the 120 multiple-choice questions of the Swiss Society of General Internal Medicine (SSGIM) board examination were derived from publicly available MKSAP questions (Medical Knowledge Self-Assessment Program®). The possibility to memorise publicly available questions may unduly influence the candidates' examination performance. Therefore, the examination board raised concerns that the examination did not meet the objective of evaluating the application of knowledge. The society decided to develop new, "Helvetic" questions to improve the examination. The aim of the present study was to quantitatively assess the degree of difficulty of the Helvetic questions (HQ) compared with publicly available and unavailable MKSAP questions and to investigate whether the degree of difficulty of MKSAP questions changed over time as their status changed from publicly available to unavailable. The November 2019 examination consisted of 40 Helvetic questions, 40 publicly available questions from MKSAP edition 17 (MKSAP-17) and 40 questions from MKSAP-15/16, which were no longer publicly available at the time of the examination. An one factorial univariate analysis of variance (ANOVA) examined question difficulty (lower values mean higher difficulty) between these three question sets. A repeated ANOVA compared the difficulty of MKSAP-15/16 questions in the November 2019 examination with the difficulty of the exact same questions from former examinations, when these questions belonged to the publicly available MKSAP edition. The publicly available MKSAP-17 and the publicly unavailable Helvetic questions served as control. The analysis of the November 2019 exam showed a significant difference in average item difficulty between Helvetic and MKSAP-17 questions (71% vs 86%, p <0.001) and between MKSAP-15/16 and MKSAP-17 questions (70% vs 86%, p <0.001). There was no significant difference in item difficulty between Helvetic and MKSAP-15/16 questions (71% vs 70%, p = 0.993). The repeated measures ANOVA on question use and the three question categories showed a significant interaction (p <0.001, partial eta-squared = 0.422). The change in the availability of MKSAP-15/16 questions had a strong effect on difficulty. Questions became on average 21.9% more difficult when they were no longer publicly available. In contrast, the difficulty of the MKSAP-17 and Helvetic questions did not change significantly across administrations. This study provides the quantitative evidence that the public availability of questions has a decisive influence on question difficulty and thus on SSGIM board examination performance. Reducing the number of publicly available questions in the examination by introducing confidential, high-quality Helvetic questions contributes to the validity of the board examination by addressing higher order cognitive skills and making rote-learning strategies less effective.

Assessment of differential item functioning across English and Spanish versions of the Orofacial Esthetic Scale.

To investigate differential item functioning across the Spanish and English versions of the Orofacial Esthetic Scale (OES). OES data were collected from a consecutive sample of a total of 622 adult dental patients (311 English- and 311 Spanish-speaking adults) from HealthPartners dental clinics in Minnesota. We inspected boundary characteristic curves (BCCs) based on item response theory (IRT) in English speakers and Spanish speakers to examine differential item functioning (DIF). Using the Samejima's graded response model, we used the item response theory log-likelihood ratio (IRTLR) approach to test whether DIF was statistically significant. Inspection of the BCCs did not reveal substantial differences in item difficulty and discrimination between English speakers and Spanish speakers. When IRTLR tests were performed for the seven OES items, no item was flagged with significant DIF after P-values were adjusted for multiple comparisons. Pooling of Orofacial Esthetic Scale (OES) data from English and Spanish-speaking dental patients for analysis and interpretation is supported by the absence of differential item functioning (DIF) across the two language groups.

The long and winding road: A comprehensive analysis of 50 years of Eysenck instruments for the assessment of personality

Instruments for the assessment of the Eysenckian superfactors of personality, Psychoticism (P), Extraversion (E), and Neuroticism (N), were developed over the course of almost 50 years. Typically the convergence with the precursor was examined when a new scale was published. In the present study the continuity and change of the substance of P, E, and N is tested by administering all instruments to a sample simultaneously, together with measures of the Five-Factor Model. A factor analysis of the 19 markers of the PEN model clearly yielded three factors, with higher loadings for E and N compared to P. The superfactors typically were measured purely after the historically second (or third, for P) instrument. Analysing the item difficulty confirmed that the P items were softened during the revisions but this created a confounding of item difficulty and content: The earlier “tough” items (mostly low Agreeableness) were gradually complemented by “softer” items representing the presumed obverse of P, superego strength (mostly low Conscientiousness). Finally, a part of the observed heterogeneity of P was due to these differences in item difficulty. Overall, the EPQ-R seems to be the most valid single measure of the PEN model.

Growing Pains: Positive Learning Outcomes but Evaluation Fallouts from Flipping the Embryology Curriculum in an Integrated Medical Gross Anatomy Course

Although embryology lays a strong foundation for understanding gross anatomy, variations, and congenital anomalies, its educational contact hours in the North American medical curricula have become minimal and integrated with other subjects. This poses challenges for students to learn the spatially complex developmental processes, leading many to seek external online resources or resort to rote memorization for the short‐term, neither an ideal solution.To promote effective learning and retention, embryology curriculum was offered in a flipped‐class format in the Fall of 2019 for first‐year medical students. Eleven, 8–16 minute tutorial videos were produced for each hour of embryology lecture in the integrated anatomy course. Before each embryology lecture, students were required to view the tutorial and complete a quiz. The lectures were customized based on pre‐lecture quiz outcomes and in‐class poll to review the difficult concepts, to solve clinical cases, and to complete additional assessments. Before each test in the course, students completed 2 additional hours of active learning sessions, which provided a variety of resources, including more clinical cases, assessment‐based learning, and various visual and physical embryo models. In an IRB‐exempt (#19‐1716), retrospective study, students’ embryology exam performances from the flipped class curriculum were compared with the previous year’s data (2018). The 2018 cohort received nearly identical educational materials, including the learning objectives, active learning sessions, and assessments, but lectures were provided in didactic presentations. The embryology instructor was the same individual in 2018 and 2019.Embryology exam data were analyzed using Item Response Theory, controlled for item difficulty differences. The results reveal that the 2019, flipped‐class cohort (n=184) had higher embryology exam performances in all 4 exams than the 2018, didactic cohort (n=184). T‐Tests revealed that the 2019 cohort performed significantly higher than the 2018 counterparts on embryology in the 2nd exam (p&lt;0.01) and the comprehensive final (p&lt;0.05). Interestingly, in student evaluations, the same embryology instructor was rated significantly lower on educational effectiveness by the 2019 cohort compared to the 2018 cohort. Thematic analysis of the evaluation comments revealed that though 2019 students found the instructor likable and appreciated the resources, efforts, and enthusiasm, the overwhelming majority disliked the flipped classroom model, citing that it did not help them learn, caused more confusion, and lectures felt disorganized.The learning outcomes of this study are consistent with the literature indicating the effectiveness of the flipped classroom in student learning and engagement; however, the negative outcomes of the educator evaluation is alarming and points to the importance of student education about evidence‐based pedagogical practices and student buy‐in.

Detecting Differential Item Functioning and Differential Step Functioning Due to Differences that Should Matter

This study illustrates the use of differential item functioning (DIF) and differential step functioning (DSF) analyses to detect differences in item difficulty that are related to experiences of examinees, such as their teachers’ instructional practices, that are relevant to the knowledge, skill, or ability the test is intended to measure. This analysis is in contrast to the typical use of DIF or DSF to detect differences related to characteristics of examinees, such as gender, language, or cultural knowledge, that should be irrelevant. Using data from two forms of Ontario’s Grade 9 Assessment of Mathematics, analyses were performed comparing groups of students defined by their teachers’ instructional practices. All constructed-response items were tested for DIF using the Mantel Chi-Square, standardized Liu Agresti cumulative common log-odds ratio, and standardized Cox’s noncentrality parameter. Items exhibiting moderate to large DIF were subsequently tested for DSF. In contrast to typical DIF or DSF analyses, which inform item development, these analyses have the potential to inform instructional practice.

Analysis of MCQ and distractor use in a large first year Health Faculty Foundation Program: assessing the effects of changing from five to four options

BackgroundMultiple choice questions are commonly used in summative assessment. It is still common practice for tertiary institutions and accrediting bodies to use five-option single best answer multiple choice questions, despite a substantial body of evidence showing that multiple choice questions with only three or four options provide effective and discriminatory assessment.MethodsIn this study we investigated the distribution of distractor efficacy in exams from four large first-year undergraduate courses in chemistry and in anatomy and physiology in a Health Faculty; assessed the impact on overall student score after changing from five-option to four-option single best answer multiple choice questions; and assessed the impact of changing from five options to four options on item difficulty and discrimination.ResultsFor the five-option questions analysed, 19% had four effective distractors, which is higher than previous studies, but still a minority of questions. After changing from five to four options, the overall student performance on all multiple choice questions was slightly lower in the second offering of one course, slightly higher in the second offering of another course, and similar in the second offering for two courses. For a subset of questions that were used in both offerings, there were negligible differences in item difficulty and item discrimination between offerings.ConclusionsThese results provide further evidence that five-option questions are not superior to four-option questions, with reduction to four options making little if any difference to overall performance, particularly when MCQ is used in conjunction with other assessment types (including short answer questions, and practical or laboratory assessment). Further areas of study that arise from these findings are: to investigate the reasons for resistance to changing established assessment practice within institutions and by accrediting bodies; and to analyse student perceptions of the impact of a reduced number of options in MCQ-based assessment.

Normalization rules for item response models: Simulation results

ABSTRACTThis research examines the statistical methodology that is used to estimate the parameters in item response models. An integral part of an item response model is the normalization rule that is used to identify the distributional parameters. The main result shown here is that only Verhelst–Glas normalizations that arbitrarily set one difficulty and one dispersion parameter to unity are consistent with the basic assumptions underlying the two-parameter logistic model. Failure to employ this type of normalization will lead to scores that depend on the item composition of the test and differential item difficulty (DIF) will compromise the validity of the estimated ability scores when different groups are being compared. It is also shown that some of the tests for DIF fail when the data are generated by an IRT model with a random effect. Most of the results are based on simulations of a four item model. Because the data generation mechanism is known, it is possible to determine the effect on ability scores and parameter estimates when different normalizations or different distribution parameter values are used.

Exploring the item features of a science assessment with complex tasks

Item explanatory models have the potential to provide insight into why certain items are easier or more difficult than others. Through the selection of pertinent item features, one can gather validity evidence for the assessment if construct-related item characteristics are chosen. This is especially important when designing assessment tasks that address new standards. Using data from the Learning Progressions in Middle School Science Instruction and Assessment (LPS) project, this paper adopts an “item explanatory” approach and investigates whether certain item features can explain differences in item difficulties by applying an extension of the linear logistic test model. Specifically, this paper explores the effects of five features on item difficulty: type (argumentation, content, embedded content), scenario-based context, format (multiple-choice or open-ended), graphics, and academic vocabulary. Interactions between some of these features were also investigated. With the exception of context, all features had a statistically significant effect on difficulty.

A Psychometric Analysis of the Italian Version of the eHealth Literacy Scale Using Item Response and Classical Test Theory Methods

BackgroundThe eHealth Literacy Scale (eHEALS) is a tool to assess consumers’ comfort and skills in using information technologies for health. Although evidence exists of reliability and construct validity of the scale, less agreement exists on structural validity.ObjectiveThe aim of this study was to validate the Italian version of the eHealth Literacy Scale (I-eHEALS) in a community sample with a focus on its structural validity, by applying psychometric techniques that account for item difficulty.MethodsTwo Web-based surveys were conducted among a total of 296 people living in the Italian-speaking region of Switzerland (Ticino). After examining the latent variables underlying the observed variables of the Italian scale via principal component analysis (PCA), fit indices for two alternative models were calculated using confirmatory factor analysis (CFA). The scale structure was examined via parametric and nonparametric item response theory (IRT) analyses accounting for differences between items regarding the proportion of answers indicating high ability. Convergent validity was assessed by correlations with theoretically related constructs.ResultsCFA showed a suboptimal model fit for both models. IRT analyses confirmed all items measure a single dimension as intended. Reliability and construct validity of the final scale were also confirmed. The contrasting results of factor analysis (FA) and IRT analyses highlight the importance of considering differences in item difficulty when examining health literacy scales.ConclusionsThe findings support the reliability and validity of the translated scale and its use for assessing Italian-speaking consumers’ eHealth literacy.

The Impact of Various Class-Distinction Features on Model Selection in the Mixture Rasch Model

ABSTRACTThe purpose of the current study is to examine the performance of four information criteria (Akaike's information criterion [AIC], corrected AIC [AICC] Bayesian information criterion [BIC], sample-size adjusted BIC [SABIC]) for detecting the correct number of latent classes in the mixture Rasch model through simulations. The simulation study manipulated various class-distinction features (percentages of class-variant items, magnitudes, and patterns of item difficulty differences) and mixing proportions, assuming that a mixture Rasch model with two latent classes was the true model. Unlike previous studies that showed BIC's superiority to other indices, our findings from this study suggested that the four information criteria had differential performance depending on the percentage of class-variant items and the magnitude and pattern of item difficulty differences under a two-class structure. Furthermore, the present study revealed that AICC and SABIC generally performed as good as or better than their counterparts, AIC and BIC, respectively, for the class-class structure with a sample of 3,000.

The Italian Catquest-9SF cataract questionnaire: translation, validation and application

BackgroundTo validate the Catquest-9SF questionnaire in Italian, assess the change in visual disability with cataract surgery and determine the correlation between pre-operative Catquest-9SF scores and Lens Opacities Classification System (LOCS) III cataract grading.MethodsProspective, questionnaire validation study. The Catquest-9SF questionnaire was forward and back translated and completed by 209 Italian patients before and three months following cataract surgery. Rasch analysis was used to assess its psychometric properties.ResultsThe Italian Catquest-9SF demonstrated ordered response categories, unidimensionality (item fit statistics range: 0.73–1.34), adequate person separation (2.04), and no differential item functioning. Mistargeting was evident with a mean difference in item difficulty and person ability of 2.04 logits but improved with inclusion of pre-operative data only. There was a statistically significant (Friedman tests, p < 0.001) median improvement in visual disability of 1.92, 3.57, 1.44 and 2.94 logits in patients undergoing first eye surgery with and without ocular comorbidity, and second eye surgery with and without ocular comorbidity respectively. There was no statistically significant difference in the improvements among the four groups (Kruskal-Wallis H test, X2(3) = 5.445, p = 0.142). There was no correlation between Catquest-9SF scores and nuclear opalescence (rs = 0.049, p = 0.478), nuclear colour (rs = 0.008, p = 0.909), cortical (rs = 0.066, p = 0.341), and posterior subcapsular components (rs = 0.048, p = 0.494).ConclusionsThe Italian Catquest-9SF demonstrated good psychometric properties and is suitable for use in Italian speaking patients. There were similar improvements in visual disability in patients undergoing first or second eye surgery, with or without ocular comorbidity. There was no correlation between pre-operative Catquest-9SF scores and LOCS III cataract grading.

Reducing the number of options on multiple-choice questions: response time, psychometrics and standard setting.

Despite significant evidence supporting the use of three-option multiple-choice questions (MCQs), these are rarely used in written examinations for health professions students. The purpose of this study was to examine the effects of reducing four- and five-option MCQs to three-option MCQs on response times, psychometric characteristics, and absolute standard setting judgements in a pharmacology examination administered to health professions students. We administered two versions of a computerised examination containing 98 MCQs to 38 Year2 medical students and 39 Year3 pharmacy students. Four- and five-option MCQs were converted into three-option MCQs to create two versions of the examination. Differences in response time, item difficulty and discrimination, and reliability were evaluated. Medical and pharmacy faculty judges provided three-level Angoff (TLA) ratings for all MCQs for both versions of the examination to allow the assessment of differences in cut scores. Students answered three-option MCQs an average of 5seconds faster than they answered four- and five-option MCQs (36seconds versus 41seconds; p=0.008). There were no significant differences in item difficulty and discrimination, or test reliability. Overall, the cut scores generated for three-option MCQs using the TLA ratings were 8 percentage points higher (p=0.04). The use of three-option MCQs in a health professions examination resulted in a time saving equivalent to the completion of 16% more MCQs per 1-hour testing period, which may increase content validity and test score reliability, and minimise construct under-representation. The higher cut scores may result in higher failure rates if an absolute standard setting method, such as the TLA method, is used. The results from this study provide a cautious indication to health professions educators that using three-option MCQs does not threaten validity and may strengthen it by allowing additional MCQs to be tested in a fixed amount of testing time with no deleterious effect on the reliability of the test scores.

Effects of Question Formats on Student and Item Performance

To determine the effect of 3 variations in test item format on item statistics and student performance. Fifteen pairs of directly comparable test questions were written to adhere to (standard scale) or deviate from (nonstandard scale) 3 specific item-writing guidelines. Differences in item difficulty and discrimination were measured between the 2 scales as a whole and for each guideline individually. Student performance was also compared between the 2 scales. The nonstandard scale was 12.7 points more difficult than the standard scale (p=0.03). The guideline to avoid "none of the above" was the only 1 of the 3 guidelines to demonstrate significance. Students scored 53.6% and 41.3% (p<0.001) of total points on the standard and nonstandard scales, respectively. Nonstandard test items were more difficult for students to answer correctly than the standard test items, provided no enhanced ability to discriminate between higher- and lower-performing students, and resulted in poorer student performance. Item-writing guidelines should be considered during test construction.

Repeated retrieval practice and item difficulty: Does criterion learning eliminate item difficulty effects?

A wealth of previous research has established that retrieval practice promotes memory, particularly when retrieval is successful. Although successful retrieval promotes memory, it remains unclear whether successful retrieval promotes memory equally well for items of varying difficulty. Will easy items still outperform difficult items on a final test if all items have been correctly recalled equal numbers of times during practice? In two experiments, normatively difficult and easy Lithuanian-English word pairs were learned via test-restudy practice until each item had been correctly recalled a preassigned number of times (from 1 to 11 correct recalls). Despite equating the numbers of successful recalls during practice, performance on a delayed final cued-recall test was lower for difficult than for easy items. Experiment 2 was designed to diagnose whether the disadvantage for difficult items was due to deficits in cue memory, target memory, and/or associative memory. The results revealed a disadvantage for the difficult versus the easy items only on the associative recognition test, with no differences on cue recognition, and even an advantage on target recognition. Although successful retrieval enhanced memory for both difficult and easy items, equating retrieval success during practice did not eliminate normative item difficulty differences.