Difficult Items Research Articles

Bias and Bias Correction Method for Nonproportional Abilities Requirement (NPAR) Tests

AbstractTest items must often be broad in scope to be ecologically valid. It is therefore almost inevitable that secondary dimensions are introduced into a test during test development. A cognitive test may require one or more abilities besides the primary ability to correctly respond to an item, in which case a unidimensional test score overestimates the primary ability and creates interpretability problems. In this article, we demonstrate the nonproportional abilities requirement, a phenomenon with which secondary abilities are more required for difficult items. A novel and practical method for correcting bias in the primary ability is proposed and illustrated using a real data set from an international assessment. Simulation data are also used to evaluate the performance of the method.

Development of multiple choice question bank in otorhinolaryngology by item analysis: a cross-sectional study

<p class="abstract"><strong>Background:</strong> Multiple choice questions (MCQs) are routinely used for formative and summative assessment in medical education. Item analysis is a process of post validation of MCQ tests, whereby items are analyzed for difficulty index, discrimination index and distractor efficiency, to obtain a range of items of varying difficulty and discrimination indices. This study was done to understand the process of item analysis and analyze MCQ test so that a valid and reliable MCQ bank in otorhinolaryngology is developed.</p><p class="abstract"><strong>Methods:</strong> 158 students of 7<sup>th</sup> Semester were given an 8 item MCQ test. Based on the marks achieved, the high achievers (top 33%, 52 students) and low achievers (bottom 33%, 52 students) were included in the study. The responses were tabulated in Microsoft Excel Sheet and analyzed for difficulty index, discrimination index and distractor efficiency. </p><p class="abstract"><strong>Results:</strong> The mean (SD) difficulty index (Diff-I) of 8 item test was 61.41% (11.81%). 5 items had a very good difficulty index (41% to 60%), while 3 items were easy (Diff-I >60%). There was no item with Diff-I <30%, i.e. a difficult item, in this test. The mean (SD) discrimination index (DI) of the test was 0.48 (0.15), and all items had very good discrimination indices of more than 0.25. Out of 24 distractors, 6 (25%) were non-functional distractors (NFDs). The mean (SD) distractor efficiency (DE) of the test was 74.62% (23.79%).</p><p class="abstract"><strong>Conclusions:</strong> Item analysis should be an integral and regular activity in each department so that a valid and reliable MCQ question bank is developed.</p>

Differential item functioning in the cognitive screener used in the Longitudinal Aging Study in India.

The Longitudinal Aging Study in India (LASI) was initiated to capture data to be comparable to the Health and Retirement Survey (HRS) and hence used study instruments from the HRS. However, a rigorous psychometric evaluation before adaptation of cognitive tests may have indicated bias due to diversities across Indian states such as education, ethnicity, and urbanicity. In the present analysis, we evaluated if items show differential item functioning (DIF) by literacy, urbanicity, and education status. We calculated proportions for each item and weighted descriptive statistics of demographic characteristics in LASI. Next, we evaluated item-level measurement differences by testing for DIF using the alignment approach implemented using Mplus software. We found that cognitive items in the LASI interview demonstrate bias by education and literacy, but not urbanicity. Items relating to animal (word) fluency show DIF. The model rates correct identification of the prime minister as the most difficult binary response item whereas the day of the week and numeracy items are rated comparatively easier. Our study would facilitate comparison across education, literacy and urbancity to support analyses of differences in cognitive status. This would help future instrument development efforts by recognizing potentially problematic items in certain subgroups.

Learning to read facilitates the retrieval of phonological representations in rapid automatized naming: Evidence from unschooled illiterate, ex-illiterate, and schooled literate adults.

Rapid automatized naming (RAN) of visual items is a powerful predictor of reading skills. However, the direction and locus of the association between RAN and reading is still largely unclear. Here, we investigated whether literacy acquisition directly bolsters RAN efficiency for objects, adopting a strong methodological design, by testing three groups of adults matched in age and socioeconomic variables, who differed only in literacy/schooling: unschooled illiterate and ex-illiterate, and schooled literate adults. To investigate in a fine-grained manner whether and how literacy facilitates lexical retrieval, we orthogonally manipulated the word-form frequency (high vs. low) and phonological neighborhood density (dense vs. spare) of the objects' names. We observed that literacy experience enhances the automaticity with which visual stimuli (e.g., objects) can be retrieved and named: relative to readers (ex-illiterate and literate), illiterate adults performed worse on RAN. Crucially, the group difference was exacerbated and significant only for those items that were of low frequency and from sparse neighborhoods. These results thus suggest that, regardless of schooling and age at which literacy was acquired, learning to read facilitates the access to and retrieval of phonological representations, especially of difficult lexical items.

The first glance is the weakest: “Tasteful” individuals are slower to judge visual art

Recent research (Myszkowski, Çelik, & Storme, 2018) has suggested that the ability to form accurate visual aesthetic judgments – an ability referred to as aesthetic sensitivity (Child, 1964), or “good taste” (Eysenck, 1983) – could be explained by the extent to which one taps into extensive processing strategies when facing aesthetic objects. Because individual differences in processing extensiveness may lead to different processing speed, we hypothesize that individuals with high visual aesthetic sensitivity present slower responding to visual aesthetic sensitivity tasks – even without time constraints. 201 adults took the Visual Aesthetic Sensitivity Test-Revised (Myszkowski & Storme, 2017), and their responses and response times were analyzed through joint hierarchical item-response theory modeling (van der Linden & Fox, 2016). As hypothesized, latent speed was negatively correlated at −0.47 (95% HPD [−0.61, −0.32]) with latent accuracy. Similar findings were obtained, accounting for guessing or not, including aberrant response patterns or not. In addition, more difficult items were also more time intensive. These findings are discussed as a substantiation that an important explanation for individual differences in visual aesthetic sensitivity lies in how much individuals are disposed to extensively process aesthetic objects.

Critical Success Factors of System Requirements Definition in IT System Development: An Industrial Case Study.

The success of IT system development is largely dependent on the System Requirements Definition (SRD) phase. Researches on Critical Success Factors (CSFs) in the SRD phase are beneficial to the success of IT system development. However, reports that consider the situation in the system requirements definition phase is lacking and these studies try to develop universal truths for CSFs without lessons learnt from empirical evidence need to be characterized. This study is a step towards bridging this gap in characterized evidence to discover “difficult items” in the SRD phase. Moreover, we conducted a case study to justify the importance of CSFs that could be 1) Customer/User Involvement, 2) Clear project goals, and 3) Technical skills of the project team in the SRD phase. The results of the characterization indicated that those major CSFs are consistent. Another issue can also be mitigated by examining Agile method.

Evaluating Quality and Reliability of Final Exam Questions for Probability and Statistics Course Using Rasch Model

Evaluation of the questions’ level of complexity for the statistical course was proposed using the revised version of Bloom’s taxonomy. The use of Bloom's taxonomy in statistical examination papers allows the degree of difficulty to be pseudo-objectively assessed. Well-constructed questions in the final examination will help in measuring students' abilities based on comprehensive cognitive skills. Therefore, this study used Rasch Model to evaluate the quality and reliability of final exam questions for probability and statistics course. According to research findings, five out of 30 questions are considered as misfit items. It is therefore recommended that these items be removed or rephrased to better suit the students’ ability level in a course. Whereas, nine questions have significant differences between taxonomy level and Rasch level that require further analysis. Overall, students view the set of exam questions as simple due to the unavailability of difficult items. Based on this result, it is suggested that the exam questions should undergo verification process from the expert and students should be exposed early to various types of questions with different level of difficulty.

Using the Rasch model to analyze the test of understanding of vectors

The test of understanding of vectors (TUV) is a multiple-choice test that was recently developed to assess student understanding of vector concepts required in introductory physics courses at the university level. In this study, the TUV was administered to 889 first-year students at the University of Zagreb. The Rasch model was used to evaluate the functioning of the TUV, and to determine the difficulty scale of vector concepts evaluated by this assessment instrument. The results of the Rasch analysis showed a good functioning of test items. However, the TUV was not well targeted to our sample of first-year engineering and science students in introductory physics courses. About half of the students had abilities outside the range of item difficulties, indicating that the TUV would benefit from including more difficult items that would help to better estimate the abilities of these students. Further analysis of the data from subpopulations in our sample showed that the TUV can be well targeted for certain student populations. Students’ scores suggest that the most difficult vector concept tested by the TUV is the unit vector, followed by the cross product, subtraction of vectors, dot product, and direction of a vector.

The Script Work in the Complex Object of Copyrights

In article the questions connected with features of the civil mode of the scenario work as a part of difficult objects of author's rights, such as audiovisual works are investigated (movies, film formats and others). The author in detail considers questions of concepts of the scenario work and a difficult object of author's rights. At the same time the special-purpose character of creation of the scenario work as object of an author's right, namely, for use as a part of difficult objects of author's rights reveals, the interconditionality of concepts, existence of two-level system of author's rights concerning a difficult object, namely, the rights for a difficult object, and the rights to results of intellectual activity which were used during creation of a difficult intellectual property item is proved.

Language mixing patterns in a bilingual individual with non-fluent aphasia

ABSTRACTBackground: Language mixing in bilingual speakers with aphasia has been reported in a number of research studies, but the reasons for the mixing and whether it reflects typical or atypical behaviour has been a matter of debate.Aims: In this study, we tested the hypothesis that language mixing behaviour in bilingual aphasia reflects lexical retrieval difficulty.Methods & procedures: We recruited a Hebrew-English bilingual participant with mild-moderate non-fluent agrammatic aphasia and assessed his languages at three timepoints. We analysed the participant’s Hebrew and English production for retrieval during single-word naming, sentences, and discourse, and identified all instances of language mixing.Outcomes & Results: We found that there was a greater frequency of language mixing during production of more difficult lexical items, namely the post-morbidly less proficient language (compared to the more proficient language), function words (compared to content words), and single-word naming (compared to retrieval in the context of connected speech tasks), but not for verbs (compared to nouns).Conclusions: In this bilingual participant with non-fluent aphasia, language mixing behaviour closely resembles lexical retrieval difficulty. Thus, we suggest that bilingual speakers with aphasia may mix their languages as a strategy to maximise communication.

An IRT-Multiple Indicators Multiple Causes (MIMIC) Approach as a Method of Examining Item Response Latency.

The analysis of response time has received increasing attention during the last decades, since evidence from several studies supported the argument that there is a direct relationship between item response time and test performance. The aim of this study was to investigate whether item response latency affects person's ability parameters, in that it represents an adaptive or maladaptive practice. To examine the above research question data from 8,475 individuals completing the computerized version of the Postgraduate General Aptitude Test (PAGAT) were analyzed. To determine the extent to which response latency affects person's ability, we used a Multiple Indicators Multiple Causes (MIMIC) model, in which every item in a scale was linked to its corresponding covariate (i.e., item response latency). We ran the MIMIC model within the Item Response Theory (IRT) framework (2-PL model). The results supported the hypothesis that item response latency could provide valuable information for getting more accurate estimations for persons' ability levels. Results indicated that for individuals who invest more time on easy items, their likelihood of success does not improve, most likely because slow and fast responders have significantly different levels of ability (fast responders are of higher ability compared to slow responders). Consequently, investing more time for low ability individuals does not prove to be adaptive. The opposite was found for difficult items: individuals spending more time on difficult items increase their likelihood of success, more likely because they are high achievers (in difficult items individuals who spent more time were of significantly higher ability compared to fast responders). Thus, it appears that there is an interaction between the difficulty of the item and person abilities that explain the effects of response time on likelihood of success. We concluded that accommodating item response latency in a computerized assessment model, can inform test quality and test takers' behavior, and in that way, enhance score measurement accuracy.

Placing Multiple Tests on a Common Scale Using a Post-test Anchor Design: Effects of Item Position and Order on the Stability of Parameter Estimates

When there is an interest in tracking longitudinal trends of student educational achievement using standardized tests, the most common linking approach generally involves the inclusion of a common set of items across adjacent test administrations. However, this approach may not be feasible in the context of high-stakes testing due to undesirable exposure of administered items. In this paper, we propose an alternative design, which allows for the equating of multiple operational tests with no items in common based on the inclusion of common items in an anchor test administered in a post-test condition. We tested this approach using data from the assessment program implemented in Italy by the National Institute for the Educational Evaluation of Instruction and Training for the years 2010-2012, and from a convenience sample of 832 8th grade students. Additionally, we investigated the impact on functioning of common items of varying item position and orders across test forms. Linking of tests was performed using multiple-group Item Response Theory modeling. Results of linking indicated that operational tests showed little variation in difficulty over the years. Investigation of item position and order effects showed that changes in item position closer to the end of the test, as well as the positioning of difficult items at the beginning or in the middle section of a test lead to a significant increase in difficulty of common items. Overall, findings indicate that this approach represents a viable linking design, which can be useful when the inclusion of common items across operational tests is not possible. The impact of differential item functioning of common items on equating error and the ability to detect ability trends is discussed.

Children’s Object Structure Perspective-Taking: Training and Assessment

Spatial abilities—required in both academic and everyday information processing—are recommended as an important target for explicit instruction in the K-12 curriculum. However, most school curricula do not address this spatial issue, probably because spatial ability is a general rather than domain-specific skill and also due to debate regarding individuals’ ability to transfer general skills to novel tasks. In particular, little is known about elementary school children’s object structure perspective-taking (OSPT) ability. We developed an age-appropriate OSPT assessment tool and examined differences in OSPT ability between 1st and 4th graders who did or did not receive OSPT training; children’s OSPT ability in relation to 3 object types: detailed everyday objects, contour only, and abstract geometrical target objects; the ability to identify a view seen by an observer and the ability to determine the vantage point from which a given view is seen; and problem-solving involving far transfer and difficult items. Problem-solving strategies and relations between these young children’s OSPT performance and their mathematic achievements were examined to further investigate the possible contribution of spatial ability to STEM learning. Our findings are promising in that a relatively short training effected significantly children’s OSPT ability—a general intellectual skill—calling for the development and introduction of OSPT-relevant K-12 curriculum, possibly using our training program as a model. Such curriculum is relevant in particular for girls, of whom OSPT showed significant correlation with mathematic scores for contoured and geometric objects. Fourth graders performed better than 1st graders, but the latter improved more.

Effects of Removing Responses With Likely Random Guessing Under Rasch Measurement on a Multiple-Choice Language Proficiency Test

ABSTRACTWith multiple options to choose from, there is always a chance of lucky guessing by examinees on multiple-choice (MC) items, thereby potentially introducing bias in item difficulty estimates. Correct responses by random guessing thus pose threats to the validity of claims made from test performance on an MC test. Under the Rasch framework, the current study investigates the effects of removing responses with likely guessing on item difficulty estimates, person ability measures, and test information function (i.e., a function of measurement precision for person ability) on an MC language proficiency test. Results show that removing responses with likely guessing leads to difficult items becoming more difficult and high-performing examinees receiving higher ability measures, because lucky guesses are more likely to appear in responses to difficult items than to easy items; therefore, removing responses with likely guessing results in lower proportion correct (i.e., higher item difficulty) for difficult items. More importantly, the current study shows that the measurement precision for high-performing examinees increases after accounting for likely random guessing, while the measurement precision for low- and medium-performing examinees remains similar with and without likely guessing. Implications for operational scoring of examinees are discussed.

Advocacy for use of the modified Iowa Level of Assistance Scale for clinical use in patients after hip replacement: an observational study

ObjectivesTo test the internal consistency and item difficulty of the modified Iowa Level of Assistance Scale (mILAS). DesignRetrospective observational study. SettingTwo orthopaedic wards of two general hospitals. ParticipantsFollowing elective primary unilateral total hip replacement surgery, all participants performed mILAS activities that were scored daily to assess their recovery of activities during hospitalisation. Main outcome measuresThe internal consistency and the level of assistance needed by the patient (item difficulty) of the mILAS were calculated using data from Deventer Hospital, Deventer, the Netherlands (n=255). A cross-validation was performed using data from Nij Smellinghe Hospital, Drachten, the Netherlands (n=224). ResultsThe internal consistency of the mILAS was acceptable on all three postoperative days (α=0.84 to 0.97). Cronbach’s α and Rasch analysis revealed a misfit of stair climbing with the other items of the mILAS. The item difficulty of the mILAS items changed over the first two postoperative days. During the first three postoperative days, the sit to supine transfer was generally the most difficult item to achieve, and the sit to stand transfer was the least difficult item to achieve as rated by physiotherapists. The cross-validation analysis revealed similar results. ConclusionsThe mILAS is a clinically sound measurement tool to assess the ability of patients to perform five functional tasks safely during hospitalisation. Stair climbing appears to be the easiest item to complete, and the sit to supine transfer is generally the most difficult after surgery.

A psychometric study of the decisional conflict scale in surrogate decision makers

ObjectiveTo assess the psychometric properties of the 16-item Decisional Conflict Scale (DCS) in surrogate decision makers. MethodsWith a sample of 472 surrogates from intensive care units in five academic medical centers across the United States, we performed the analysis in five phases to 1) model the congeneric structure with confirmatory factor analysis and assess 2) internal consistency reliability, 3) the unidimensional or global assessment, 4) factorial invariance across surrogate gender, and 5) individual item influence on the domains. ResultsThe congeneric model fit the data, with all factor loadings (0.577–0.955) statistically significant at p < 0.05. All subdomains had acceptable internal consistency (0.751–0.981). The bifactor model supported the sub-domains or the global construct as appropriate measurement models. The DCS demonstrated invariance for use across surrogate genders. The most difficult item for surrogates to complete was “This decision is easy for me to make” [MNSQ Infit/Outfit: 2.37/3.27] and the easiest item was “I have enough advice to make a choice” [0.70/0.67]. ConclusionsOverall, the DCS demonstrated good fit, and can be considered a valid and reliable tool to use with the surrogate population. Practice ImplicationsMeasuring surrogate decisional conflict could be especially useful to determine how to improve the decision-making process.

Sequential Difficulty Effects in Cognitive and Sensorimotor Tasks: Insights from Arithmetic and Fitts Tasks

Abstract The present study tested sequential difficulty effects (SDEs) in arithmetic problem solving and Fitts’ aiming task for the same individuals. SDEs refer to poorer performance on current items following harder items relative to after easier items. Young and older adults accomplished an arithmetic task (i.e., finding the approximate products to two-digit multiplication problems) and a Fitts aiming task (i.e., performing rapid pointing movements to reach the finish areas). Current items were preceded by two easy or difficult items (i.e., in the repeated-precursor condition) or only one easy or difficult item (i.e., in the unrepeated-precursor condition). Participants’ performance revealed SDEs in both the arithmetic and the aiming tasks only when the difficulty level of the precursor items was repeated. Data also revealed comparable SDEs in both age groups during the arithmetic task but SDEs only in older adults while participants accomplished the aiming task. These findings have a number of implications for our understanding of mechanisms underlying SDEs and age-related differences in SDEs, as they suggest that SDEs involve both domain-general and domain-specific mechanisms that are differentially influenced by aging.

영어 파생어의 형태 · 통사적 특성과 출현 빈도가 학습 난이도에 미치는 영향

This study investigated Korean middle school students’ acquisition of high-frequency derived words in English. A total of 36 pairs of base and derived word forms within the band of 1,000 most frequent words were extracted from the Corpus of Contemporary American English, and then a test set was developed to assess students’ morphological awareness and their ability to choose the correct word form in a given syntactic context. The main findings are as follows: First, Korean students tend to acquire the V-N and Adj-N types of derivation better than the N-Adj and Adj-Adv ones. Second, base forms are not necessarily acquired better and earlier than derived forms. Third, the productivity of a suffix does not predict its learning difficulty to a significant degree. Despite its high frequency and productivity, the suffix –al turns out to be the most difficult item for Korean students. Fourth, acquisition seems to be more facilitated when the two morphologically related forms show a moderate degree of difference in frequency.

A psychometric analysis of a newly developed summative, multiple choice question assessment adapted from Canada to a Middle Eastern context

IntroductionAccreditation necessitates that assessment methods reflect the standards established by the accrediting body. The process of adapting assessments to a new context can present unique challenges with uncertainty around psychometric defensibility of the adapted exam. MethodsA psychometric analysis of a summative multiple-choice-question (MCQ) assessment, adapted from Canada, for graduating pharmacy students from a Canadian accredited program in Qatar was conducted. Rates of difficult items, item discrimination measured by point biserial correlation (rpb), and non-functioning distractors (NFDs) were calculated to identify deficiencies and challenges with an adapted exam. Challenges encountered throughout the adaption process and recommendations were documented. ResultsOverall score of a 90-item, four option, MCQ exam ranged from 46.7% to 78.9% (mean of 61.9%). For difficulty, there were 17 items with less than 30% of students answering correctly, while 29 items had unacceptable or poor discrimination (rpb < 0.1). NFDs occurred in 78 items with 49 containing at least two NFDs. Discussion and conclusionsDifficulty of the exam was deemed acceptable yet discriminator ability requires improvement. The high frequency of questions with NFDs suggests that faculty have difficulty developing plausible distractors for an adapted MCQ exam. This could be due to a lack of training or requirement for inclusion of too many distractor options. While it is feasible to implement an assessment adapted from a different learning environment, measures need to be taken to improve psychometric defensibility. The high number of questions with NFDs indicates that the current method of exam development does not encourage the incorporation of functional distractors.

An Item Analysis of English Summative Test for The First Semester of The Third Grade Junior High School Students in Surakarta

&lt;p&gt;The aims of this research are:(1) to find out whether the multiple-choice items of English summative test for the third grade junior high school students in Surakarta have fulfilled the criteria of good test or not; and (2)To describe whether the multiple-choice items of English summative test for the third grade junior high school students in Surakarta have fulfilled the criteria of good test item viewed from difficulty level, discrimination power, and distractor evaluation or not. The data were taken from 100 students‟ answer sheets in five schools in Surakarta. The test item was analyzed by using item analysis technique seen from the score and the language content analysis. The analysis was considered by using three aspects i.e. index of discriminating power, level of difficulty and the distractor evaluation. The research finding shows that (1) the difficulty level shows 13% for very difficult item, 22% for very easy item, and 64% for satisfactory item; (2) only 35% of all test items have appropriate index of discriminating power; and (3) only 49% of the distractors are effective since they were selected by the students. Based on the three criteria of a good test item above, there are only 46% of multiple choice items fulfilling the criteria of a good test items.&lt;/p&gt;