Easy Items Research Articles

A Method for Detecting Regression of Hard and Easy Item Angoff Ratings

AbstractOne common phenomenon in Angoff standard setting is that panelists regress their ratings in toward the middle of the probability scale. This study describes two indices based on taking ratios of standard deviations that can be utilized with a scatterplot of item ratings versus expected probabilities of success to identify whether ratings are regressed in toward the middle of the probability scale. Results from a simulation study show that the standard deviation ratio indices can successfully detect ratings for hard and easy items that are regressed in toward the middle of the probability scale in Angoff standard‐setting data, where previously proposed indices often do not work as well to detect these effects. Results from a real data set show that, while virtually all raters improve from Round 1 to Round 2 as measured by previously developed indices, the standard deviation ratios in conjunction with a scatterplot of item ratings versus expected probabilities of success can identify individuals who may still be regressing their ratings in toward the middle of the probability scale even after receiving feedback. The authors suggest using the scatterplot along with the standard deviation ratio indices and other statistics for measuring the quality of Angoff standard‐setting data.

Exploring the Influence of Judge Proficiency on Standard‐Setting Judgments

AbstractSetting performance standards is a judgmental process involving human opinions and values as well as technical and empirical considerations. Although all cut score decisions are by nature somewhat arbitrary, they should not be capricious. Judges selected for standard‐setting panels should have the proper qualifications to make the judgments asked of them; however, even qualified judges vary in expertise and in some cases, such as highly specialized areas or when members of the public are involved, it may be difficult to ensure that each member of a standard‐setting panel has the requisite expertise to make qualified judgments. Given the subjective nature of these types of judgments, and that a large part of the validity argument for an exam lies in the robustness of its passing standard, an examination of the influence of judge proficiency on the judgments is warranted. This study explores the use of the many‐facet Rasch model as a method for adjusting modified Angoff standard‐setting ratings based on judges’ proficiency levels. The results suggest differences in the severity and quality of standard‐setting judgments across levels of judge proficiency, such that judges who answered easy items incorrectly tended to perceive them as easier, but those who answered correctly tended to provide ratings within normal stochastic limits.

Seeing is believing: Task-exposure specificity and the development of mathematics self-efficacy evaluations.

We examine the relevance of task exposure specificity in the development of self-efficacy evaluations among 15-year-old students in Organisation for Economic Co-operation and Development (OECD) countries. We use data from the 2012 edition of the Programme for International Student Assessment (PISA) to assess if students’ self-efficacy evaluations with respect to a set of mathematics tasks are associated with how frequently they encountered those same tasks or tasks requiring them to apply similar processes or procedures. We find evidence of task exposure specificity in the development of self-efficacy evaluations toward both applied and pure mathematics problems. The relationship between task exposure and self-efficacy evaluations is not moderated by students’ socioeconomic status (SES) or anxiety toward mathematics. Exposure to easy items appears to be more strongly associated with the development of self-efficacy among high achieving students and exposure to pure mathematics problems is more strongly associated with the development of self-efficacy among girls. (PsycINFO Database Record (c) 2019 APA, all rights reserved)

Measuring Widening Proficiency Differences in International Assessments: Are Current Approaches Enough?

AbstractParticipation in international large‐scale assessments has grown over time with the largest, the Programme for International Student Assessment (PISA), including more than 70 education systems that are economically and educationally diverse. To help accommodate for large achievement differences among participants, in 2009 PISA offered low‐performing systems the option of including an easier set of items in the assessment with an aim of providing improved achievement estimates. However, there remains a lack of evidence on the performance of this design innovation. As such, we simulate a design that closely mirrors the PISA 2015 math assessment in order to empirically examine the benefits of including easy items for low‐performing countries. We extend the PISA design to include increased numbers of easy items and items that are easier than currently implemented. Findings show that the current PISA approach provides little advantage compared to a common test for all participants. Our study also demonstrates persistent bias, low coverage rates, and low correlations between generating and estimated proficiency under current designs. Through our simulation we also show that to improve achievement estimation for low performers about half of the items would need to be made significantly easier.

Short Danish Version of the Tools for Early Assessment in Math (TEAM) for 3–6-Year-Olds

ABSTRACTSound assessment tools are needed to evaluate effects of mathematics interventions that familiarize children with early mathematics concepts before they enter the formal school system. We developed a short version of an existing early mathematics tool based on analyses of data collected in a nationally representative Danish sample. Research findings: The Danish adaptation and development process of the Tools for Early Assessment in Math (TEAM) for children aged 3−6 years was carried out in four steps: (a) choosing and translating relevant items, (b) conducting a pilot study, (c) testing items in a representative sample of Danish children aged 3−6 years (n = 5,621), and (d) analyses based on Rasch models. The process resulted in a final 19-item version—the DK-TEAM (final)—that has no differential item functioning relative to age and gender and is sensitive to the full range of abilities. The great majority of the children viewed the test as enjoyable. Practice or Policy: The DK-TEAM (final) appears to be broadly applicable for young Danish children, though the modest reliability at 3 years (which may be remediable by adding easy items) should be kept in mind.

An IRT-Multiple Indicators Multiple Causes (MIMIC) Approach as a Method of Examining Item Response Latency.

The analysis of response time has received increasing attention during the last decades, since evidence from several studies supported the argument that there is a direct relationship between item response time and test performance. The aim of this study was to investigate whether item response latency affects person's ability parameters, in that it represents an adaptive or maladaptive practice. To examine the above research question data from 8,475 individuals completing the computerized version of the Postgraduate General Aptitude Test (PAGAT) were analyzed. To determine the extent to which response latency affects person's ability, we used a Multiple Indicators Multiple Causes (MIMIC) model, in which every item in a scale was linked to its corresponding covariate (i.e., item response latency). We ran the MIMIC model within the Item Response Theory (IRT) framework (2-PL model). The results supported the hypothesis that item response latency could provide valuable information for getting more accurate estimations for persons' ability levels. Results indicated that for individuals who invest more time on easy items, their likelihood of success does not improve, most likely because slow and fast responders have significantly different levels of ability (fast responders are of higher ability compared to slow responders). Consequently, investing more time for low ability individuals does not prove to be adaptive. The opposite was found for difficult items: individuals spending more time on difficult items increase their likelihood of success, more likely because they are high achievers (in difficult items individuals who spent more time were of significantly higher ability compared to fast responders). Thus, it appears that there is an interaction between the difficulty of the item and person abilities that explain the effects of response time on likelihood of success. We concluded that accommodating item response latency in a computerized assessment model, can inform test quality and test takers' behavior, and in that way, enhance score measurement accuracy.

The easy-to-hard training advantage with real-world medical images

Many medical professions require practitioners to perform visual categorizations in domains such as radiology, dermatology, and neurology. However, acquiring visual expertise is tedious and time-consuming and the perceptual strategies mediating visual categorization skills are poorly understood. In this paper, the Ease algorithm was developed to predict an item’s categorization difficulty (Ease value) based on the item’s perceptual similarity to all within-category items versus between-category items in the dataset. In this study, Ease values were used to construct an easy-to-hard and hard-to-easy training schedule for teaching melanoma diagnosis. Whereas previous visual training studies suggest that an easy-to-hard schedule benefits learning outcomes, no studies to date have demonstrated the easy-to-hard advantage with complex, real-world images. In our study, 237 melanoma and benign images were collected for training and testing purposes. The diagnostic accuracy of images was verified by an expert dermatologist. Based on their Ease values, the items were grouped into easy, medium, and hard categories, each containing an equal number of melanoma and benign lesions. During training, participants categorized images of skin lesions as either benign or melanoma and were given corrective feedback after each trial. In the easy-to-hard training condition, participants learned to categorize all the easy items first, followed by the medium items, and finally the hard items. Participants in the hard-to-easy training condition learned items in the reverse order. Post-training results showed that training in both conditions transferred to the classification of new melanoma and benign images. Participants in the easy-to-hard condition showed modest advantages both in the acquisition and retention of the melanoma diagnosis skills, but neither scheduling condition exhibited a gross advantage. The Ease values of the items predicted categorization accuracy after, but not before training, suggesting that the Ease algorithm is a promising tool for optimizing medical training in visual categorization.

Effects of Removing Responses With Likely Random Guessing Under Rasch Measurement on a Multiple-Choice Language Proficiency Test

ABSTRACTWith multiple options to choose from, there is always a chance of lucky guessing by examinees on multiple-choice (MC) items, thereby potentially introducing bias in item difficulty estimates. Correct responses by random guessing thus pose threats to the validity of claims made from test performance on an MC test. Under the Rasch framework, the current study investigates the effects of removing responses with likely guessing on item difficulty estimates, person ability measures, and test information function (i.e., a function of measurement precision for person ability) on an MC language proficiency test. Results show that removing responses with likely guessing leads to difficult items becoming more difficult and high-performing examinees receiving higher ability measures, because lucky guesses are more likely to appear in responses to difficult items than to easy items; therefore, removing responses with likely guessing results in lower proportion correct (i.e., higher item difficulty) for difficult items. More importantly, the current study shows that the measurement precision for high-performing examinees increases after accounting for likely random guessing, while the measurement precision for low- and medium-performing examinees remains similar with and without likely guessing. Implications for operational scoring of examinees are discussed.

The benefits of retrieval practice depend on item difficulty and intelligence.

The authors examined whether individual differences in fluid intelligence (gF) modulate the testing effect. Participants studied Swahili-English word pairs and repeatedly studied half the pairs or attempted retrieval, with feedback, for the remaining half. Word pairs were easy or difficult to learn. Overall, participants showed a benefit of testing over restudy. However, almost 1/3 of the sample had a negative testing effect and benefitted more from restudy than testing, as well as performing better overall. These individuals self-reported less use of shallower encoding strategies than positive testing effect participants but did not differ in other dimensions. For individuals with a positive testing effect, difficulty had differential effects on participants who scored high or low on a measure of gF, with high gF participants showing larger testing effects for difficult over easy items, whereas low gF participants showed the opposite. Working memory performance was not related to the magnitude of the testing effect; however, vocabulary knowledge revealed a similar pattern as gF, with higher vocabulary associated with a testing effect for difficult but not easy items. This suggests that the benefit of retrieval practice varies with item difficulty and participant abilities. Thus, recommendations to engage in retrieval practice should take into consideration the interactive effects of to-be-learned materials and individual differences in the learners. (PsycINFO Database Record

Equating Angoff Standard-Setting Ratings with the Rasch Model

ABSTRACTA key part of determining cut-scores when performing Angoff standard setting is utilizing equating methods to place standard-setting ratings onto the scale used to report scores to examinees. This article describes three equating methods that can be employed to place Angoff ratings onto the scale used to report scores to examinees when applying the Rasch model. It is shown that the three methods make different assumptions and that the commonly observed situation of raters regressing Angoff standard-setting ratings toward the middle of the probability scale can impact the methods in different ways. Results from a simulation study suggest that equating standard-setting ratings at the individual panelist level may better reduce the impact of raters regressing their ratings toward the middle of the probability scale for hard and easy items. Analyses of data from two standard settings suggest that there can be important differences in cut-scores when the three methods are applied in practice. The study concludes with practical considerations related to using the three equating methods to determine cut-scores.

An analysis of multiple choice questions (MCQs): Item and test statistics from mathematics assessments in senior high school

The multiple-choice test is a common test format used in education. One of the purposes of this test is to evaluate the success of the learning process in a particular subject. Therefore, the efficiency of the evaluation depends on the quality of the test items used. This research was conducted in order to reveal the quality of the final mathematics examination items statistically. It was descriptive quantitative research employing two-parameter logistic (2pl) model of Item Response Theory (IRT). The data were obtained from the sample of 353 students established using the purposive sampling technique. This finding shows that 40% of the 35 items tested are very difficult, 60% are in the medium level, and there is no easy item. The most difficult material is the trigonometric calculation. The percentage of the item discrimination index is described as follows: 8.57% of the items are categorized as very low, 51.43% are categorized as low, 31.43% of the items have a medium item discrimination index value, 5.71% have a high item discrimination index value, and 2.86% of the items are categorized as very high. Moreover, the research found that all distractors functioned well. The highest information on ability θ = 0.4 with information function value of 5.38 and SEM = 0.6. This test is suitable for students with the ability of -1.42 θ 2.65.

Sequential Difficulty Effects in Cognitive and Sensorimotor Tasks: Insights from Arithmetic and Fitts Tasks

Abstract The present study tested sequential difficulty effects (SDEs) in arithmetic problem solving and Fitts’ aiming task for the same individuals. SDEs refer to poorer performance on current items following harder items relative to after easier items. Young and older adults accomplished an arithmetic task (i.e., finding the approximate products to two-digit multiplication problems) and a Fitts aiming task (i.e., performing rapid pointing movements to reach the finish areas). Current items were preceded by two easy or difficult items (i.e., in the repeated-precursor condition) or only one easy or difficult item (i.e., in the unrepeated-precursor condition). Participants’ performance revealed SDEs in both the arithmetic and the aiming tasks only when the difficulty level of the precursor items was repeated. Data also revealed comparable SDEs in both age groups during the arithmetic task but SDEs only in older adults while participants accomplished the aiming task. These findings have a number of implications for our understanding of mechanisms underlying SDEs and age-related differences in SDEs, as they suggest that SDEs involve both domain-general and domain-specific mechanisms that are differentially influenced by aging.

An Item Analysis of English Summative Test for The First Semester of The Third Grade Junior High School Students in Surakarta

<p>The aims of this research are:(1) to find out whether the multiple-choice items of English summative test for the third grade junior high school students in Surakarta have fulfilled the criteria of good test or not; and (2)To describe whether the multiple-choice items of English summative test for the third grade junior high school students in Surakarta have fulfilled the criteria of good test item viewed from difficulty level, discrimination power, and distractor evaluation or not. The data were taken from 100 students‟ answer sheets in five schools in Surakarta. The test item was analyzed by using item analysis technique seen from the score and the language content analysis. The analysis was considered by using three aspects i.e. index of discriminating power, level of difficulty and the distractor evaluation. The research finding shows that (1) the difficulty level shows 13% for very difficult item, 22% for very easy item, and 64% for satisfactory item; (2) only 35% of all test items have appropriate index of discriminating power; and (3) only 49% of the distractors are effective since they were selected by the students. Based on the three criteria of a good test item above, there are only 46% of multiple choice items fulfilling the criteria of a good test items.</p>

Evaluation of three patient reported outcome measures following operative fixation of closed ankle fractures

BackgroundSeveral patient reported outcome measures (PROMs) are available for assessing the outcomes of ankle fracture but few have been compared for recommended measurement properties. This study compares the measurement properties of the Lower Extremity Function Scale (LEFS), Olerud Molander Ankle Score (OMAS) and Self-Reported Foot and Ankle Score (SEFAS) following ankle surgery.MethodsThe retrospective cohort study included 959 patients aged 18 years and over who underwent surgical treatment (ORIF) for unstable and closed ankle fractures in SE Norway. The PROMs were included in a postal questionnaire sent to patients’ homes in 2015, three years after surgery. Missing data, structural validity, internal consistency, test-retest reliability and validity were assessed.ResultsConfirmatory factor analysis results showed model fit for the SEFAS and a bi-dimensional LEFS with scales of easy and difficult items. The OMAS performed less satisfactorily. Cronbach’s alpha and test-retest correlations ranged from 0.82 to 0.96 and 0.91 to 0.93 respectively. The smallest detectable differences for group and individual comparisons were 14.1 to 20.6 and 0.93 to 1.55; SEFAS performed best. As hypothesised, instrument scores were highly correlated and with those for the EQ-5D and SF-36 physical functioning. Mean imputation where half or more items are completed increased usable scores by 1.4–15.7% without affecting measurement properties.ConclusionsThe three instruments largely performed satisfactorily in relation to important measurement properties but the LEFS had evidence for two dimensions relating to easier and more difficult aspects of function. Mean imputation where half or more items are completed increased the number of usable responses for all three instruments. The three instruments represent different approaches to measuring outcomes and their content should be considered carefully when choosing between them. The SEFAS is designed for a range of foot disorders including ankle fractures and has the best measurement properties in this population.

Standard setting in Australian medical schools

BackgroundStandard setting of assessment is critical in quality assurance of medical programs. The aims of this study were to identify and compare the impact of methods used to establish the passing standard by the 13 medical schools who participated in the 2014 Australian Medical Schools Assessment Collaboration (AMSAC).MethodsA survey was conducted to identify the standard setting procedures used by participating schools. Schools standard setting data was collated for the 49 multiple choice items used for benchmarking by AMSAC in 2014. Analyses were conducted for nine schools by their method of standard setting and key characteristics of 28 panel members from four schools.ResultsSubstantial differences were identified between AMSAC schools that participated in the study, in both the standard setting methods and how particular techniques were implemented. The correlation between the item standard settings data by school ranged from − 0.116 to 0.632. A trend was identified for panel members to underestimate the difficulty level of hard items and overestimate the difficulty level of easy items for all methods. The median derived cut-score standard across schools was 55% for the 49 benchmarking questions. Although, no significant differences were found according to panel member standard setting experience or clinicians versus scientists, panel members with a high curriculum engagement generally had significantly lower expectations of borderline candidates (p = 0.044).ConclusionThis study used a robust assessment framework to demonstrate that several standard setting techniques are used by Australian medical schools, which in some cases use different techniques for different stages of their program. The implementation of the most common method, the Modified Angoff standard setting approach was found to vary markedly. The method of standard setting used had an impact on the distribution of expected minimally competent student performance by item and overall, with the passing standard varying by up to 10%. This difference can be attributed to the method of standard setting because the ASMSAC items have been shown over time to have consistent performance levels reflecting similar cohort ability. There is a need for more consistency in the method of standard setting used by medical schools in Australia.

Item Analysis of Multiple Choice Questions at the Department of Paediatrics, Arabian Gulf University, Manama, Bahrain.

The current study aimed to carry out a post-validation item analysis of multiple choice questions (MCQs) in medical examinations in order to evaluate correlations between item difficulty, item discrimination and distraction effectiveness so as to determine whether questions should be included, modified or discarded. In addition, the optimal number of options per MCQ was analysed. This cross-sectional study was performed in the Department of Paediatrics, Arabian Gulf University, Manama, Bahrain. A total of 800 MCQs and 4,000 distractors were analysed between November 2013 and June 2016. The mean difficulty index ranged from 36.70-73.14%. The mean discrimination index ranged from 0.20-0.34. The mean distractor efficiency ranged from 66.50-90.00%. Of the items, 48.4%, 35.3%, 11.4%, 3.9% and 1.1% had zero, one, two, three and four nonfunctional distractors (NFDs), respectively. Using three or four rather than five options in each MCQ resulted in 95% or 83.6% of items having zero NFDs, respectively. The distractor efficiency was 91.87%, 85.83% and 64.13% for difficult, acceptable and easy items, respectively (P <0.005). Distractor efficiency was 83.33%, 83.24% and 77.56% for items with excellent, acceptable and poor discrimination, respectively (P <0.005). The average Kuder-Richardson formula 20 reliability coefficient was 0.76. A considerable number of the MCQ items were within acceptable ranges. However, some items needed to be discarded or revised. Using three or four rather than five options in MCQs is recommended to reduce the number of NFDs and improve the overall quality of the examination.

The Quality of English Final Test at the Second Semester of Third Grade Students of SMAN 1 Pagaran in Academic Year 2016/2017

The objective of this study is to find out the quality of the English final test designed for the second semester of third grade students of SMAN 1 Pagaran in academic year 2016/2017. It describes whether or not the test items have good characteristic of test in terms of validity, reliability, difficulty level, and discriminating power. The test consists of 35 items multiple choice forms. The research design uses in this study was Descriptive Qualitative Research. To find out the discriminating power of the test, the writer chose the top 31% for the upper group and top 31% for the lower group. The result of this study shows that there are 18 (51%) acceptable items to meet the criteria of validity and 17 items (49%) is Invalid. The test is reliable since has 0.676 the level of validity. The test has unacceptable index of difficulty since has 15 items (43%) too difficult and are only 5 items (14%) easy items. Whereas for discriminating power index, the writer found there are 7 (20%) has negative result of the point have to be discard, 6 (17%) poor items, 8 (22%) satisfactory items, 13 (38%) good items, and 1 (3 %) excellent item. In conclusion, English final test designed for the second semester of third grade students of SMAN 1 Pagaran in academic year 2016/2017 does not meet the criteria of effective and acceptable test.Keywords: Validity, Reliability, Level of Difficulty, Discrimination Power

Initial training with difficult items does not facilitate category learning

In the phenomenon of transfer along a continuum (TAC), initial training on easy items facilitates later learning of a harder discrimination. TAC is a widely replicated cross-species phenomenon that is well predicted by certain kinds of associative theory. A recent report of an approximately opposite phenomenon (i.e., facilitation by initial training on hard items) poses a puzzle for such theories, but is predicted by a dual-system model (COVIS). However, across four experiments, we present substantial evidence that this counterintuitive finding was in error. Rather, the result appears to be a false positive and, as such, should not form part of the evidence base for COVIS nor be considered as a counter-example to the pervasive TAC phenomenon.

Agency attributions of mental effort during self-regulated learning.

Previous results suggest that the monitoring of one's own performance during self-regulated learning is mediated by self-agency attributions and that these attributions can be influenced by poststudy effort-framing instructions. These results pose a challenge to the study of issues of self-agency in metacognition when the objects of self-regulation are mental operations rather than motor actions that have observable outcomes. When participants studied items in Experiment 1 under time pressure, they invested greater study effort in the easier items in the list. However, the effects of effort framing were the same as when learners typically invest more study effort in the more difficult items: Judgments of learning (JOLs) decreased with effort when instructions biased the attribution of effort to nonagentic sources but increased when they biased attribution to agentic sources. However, the effects of effort framing were constrained by parameters of the study task: Interitem differences in difficulty constrained the attribution of effort to agentic regulation (Experiment 2) whereas interitem differences in the incentive for recall constrained the attribution of effort to nonagentic sources (Experiment 3). The results suggest that the regulation and attribution of effort during self-regulated learning occur within a module that is dissociated from the learner's superordinate agenda but is sensitive to parameters of the task. A model specifies the stage at which effort framing affects the effort-JOL relationship by biasing the attribution of effort to agentic or nonagentic sources. The potentialities that exist in metacognition for the investigation of issues of self-agency are discussed.

Regression Effects in Angoff Ratings: Examples from Credentialing Exams

ABSTRACTThis article discusses regression effects that are commonly observed in Angoff ratings where panelists tend to think that hard items are easier than they are and easy items are more difficult than they are in comparison to estimated item difficulties. Analyses of data from two credentialing exams illustrate these regression effects and the persistence of these regression effects across rounds of standard setting, even after panelists have received feedback information and have been given the opportunity to discuss their ratings. Additional analyses show that there tended to be a relationship between the average item ratings provided by panelists and the standard deviations of those item ratings and that the relationship followed a quadratic form with peak variation in average item ratings found toward the middle of the item difficulty scale. The study concludes with discussion of these findings and what they may imply for future standard settings.