Item Errors Research Articles

Screening for technical flaws in multiple-choice items. A generalizability study.

Construction errors in multiple-choice items are quite prevalent and constitute threats to test validity of multiple-choice tests. Currently very little research on the usefulness of systematic item screening by local review committees before test administration seem to exist. The aim of this study was therefore to examine validity and feasibility aspects of review committee screening for item flaws. We examined the reliability of item reviewers’ independent judgments of the presence/absence of item flaws with a generalizability study design and found only moderate reliability using five reviewers. Statistical analyses of actual exam scores could be a more efficient way of identifying flaws and improving average item discrimination of tests in local contexts. The question of validity of human judgments of item flaws is important - not just for sufficiently sound quality assurance procedures of tests in local test contexts - but also for the global research on item flaws.

중국인 학습자의 부사격 조사 ‘에’의 대치 오류 연구 - 학습자 말뭉치를 대상으로 -

The purpose of this paper is to investigate the usage patterns and errors of the adverbial particle. We will focus on the research used in place of ‘에’, according to how it appears for various semantic items of ‘에’. In this paper, we try to investigate the errors of ‘에’, which is the survey of the Chinese Korean learners, through large scale learner data. The specific research goals are as follows. First, we try to analyze the substitution error of the adverbial particle. Second, we try to examine the pattern of confrontation error according to the dictionary meaning of ‘에’. The results of the substitution error analysis are as follows. First, the errors of substitution of ‘에’ were in the order of ‘에서’ to ‘을/를’ to ‘의’ to ‘이/가’. Second, there was a difference in the aspect of the survey which was replaced by the meaning of ‘에’. And based on the example sentences of the errors, we could estimate the cause of some errors by estimating the influence of the mother tongue through the contrast analysis. First, there is a certain Chinese semantic counterpart, but a negative transition occurs. Second, there are a lot of Chinese correspondence, and there is interference in some cases. Third, there is no corresponding expression in Chinese. Next, it is difficult to see it as a mother tongue interference, but there are some complicated errors in ‘에’ itself. It is meaningful to examine the pattern of errors in large-scale empirical data and to diagnose errors in semantic items of Chinese learners through concrete examples.

The (lack of) effect of dynamic visual noise on the concreteness effect in short-term memory

ABSTRACTIt has been suggested that the concreteness effect in short-term memory (STM) is a consequence of concrete words having more distinctive and richer semantic representations. The generation and storage of visual codes in STM could also play a crucial role on the effect because concrete words are more imaginable than abstract words. If this were the case, the introduction of a visual interference task would be expected to disrupt recall of concrete words. A Dynamic Visual Noise (DVN) display, which has been proven to eliminate the concreteness effect on long-term memory (LTM), was presented along encoding of concrete and abstract words in a STM serial recall task. Results showed a main effect of word type, with more item errors in abstract words, a main effect of DVN, which impaired global performance due to more order errors, but no interaction, suggesting that DVN did not have any impact on the concreteness effect. These findings are discussed in terms of LTM participation through redintegration processes and in terms of the language-based models of verbal STM.

The brain and the subjective experience of time. A voxel based symptom-lesion mapping study

The aim of the study was to identify the anatomical bases involved in the subjective experience of time, by means of a voxel based symptom-lesion mapping (VLSM) study on patients with focal brain damage. Thirty-three patients (nineteen with right-hemisphere lesions −RBD, and fourteen with left lesion- LBD) and twenty-eight non-neurological controls (NNC) underwent the semi-structured QUEstionnaire for the Subjective experience of Time (QUEST) requiring retrospective and prospective judgements on self-relevant time intervals. All participants also completed tests to assess general cognitive functioning and two questionnaires to evaluate their emotional state. Both groups of brain-damaged patients achieved significantly different scores from NNC on the time performance, without differences between RBD and LBD. VLSM showed a cluster of voxels located in the right inferior parietal lobule significantly related to errors in the prospective items. The lesion subtraction analysis revealed two different patterns possibly associated with errors in the prospective items (the right inferior parietal cortex, rolandic operculum and posterior middle temporal gyrus) and in the retrospective items (superior middle temporal gyrus, white matter posterior to the insula).

Mapping the brain network of the phonological loop.

The cortical and subcortical neural correlates underlying item and order information in verbal short-term memory (STM) were investigated by means of digit span in 29 patients with direct electrical stimulation during awake surgery for removal of a neoplastic lesion. Stimulation of left Broca's area interfered with span, producing significantly more item than order errors, as compared to the stimulation of the supramarginal/angular gyrus, which also interfered with span but, conversely, produced more order than item errors. Similarly, stimulation of the third segment of the left superior longitudinal fasciculus (SLF-III), also known as anterior segment of the arcuate fascicle (AF), produced more order than item errors. Therefore, we obtained two crucial results: first, we were able to distinguish between content and order information storage. Second, we demonstrated that the SLF-III is involved in transferring order information from Geschwind's area to Broca's area. In a few patients, we demonstrated that also order information of nonverbal material was disrupted by left supramarginal gyrus stimulation. Order information is thus likely stored in the supramarginal gyrus, possibly independently from the nature of the material. Hum Brain Mapp 38:3011-3024, 2017. © 2017 Wiley Periodicals, Inc.

Disentangling the effects of word frequency and contextual diversity on serial recall performance.

Research shows that contextual diversity (CD; the number of different contexts in which a word appears within a corpus) constitutes a better predictor of reading performance than word frequency (WF), that it mediates the access to lexical representations, and that controlling for contextual CD abolishes the effect of WF in lexical decision tasks. Despite the theoretical relevance of these findings for the study of serial memory, it is not known how CD might affect serial recall performance. We report the first independent manipulation of CD and WF in a serial recall task. Experiment 1 revealed better performance for low CD and for high WF words independently. Both effects affected omissions and item errors, but contrary to past research, word frequency also affected order errors. These results were confirmed in two more experiments comparing pure and alternating lists of low and high CD (Experiment 2) or WF (Experiment 3). The effect of CD was immune to this manipulation, while that of WF was abolished in alternating lists. Altogether the findings suggest a more difficult episodic retrieval of item information for words of high CD, and a role for both item and order information in the WF effect.

Phonological processing in children with specific language impairment with and without reading difficulties.

Specific language impairment (SLI) is heterogeneous and identifying subgroups within it may help explain the aetiology of the condition. Phonological processing abilities distinguish between children with SLI who do and do not have reading decoding impairments (RDIs). To probe different levels of phonological processing in children with SLI with and without RDI to investigate the cognitive basis of these differences. A total of 64 children aged 5-17 years were classified using the results of standardized language and single-word reading tests into those with no SLI and no RDI (No SLI/No RDI) (N = 18), no SLI but with RDI (No SLI/RDI) (N = 4, not included in analyses because of the small number), SLI/No RDI (N = 20), and SLI/RDI (N = 22). The groups were compared on a range of tasks engaging different levels of phonological processing (input and output processing and phonological awareness). The SLI/RDI group was distinguished from the SLI/No RDI and No SLI/No RDI groups by more errors in the longer items in non-word repetition and by poorer phonological awareness. Non-word discrimination scores indicated a gradient of performance across groups that was not associated with a qualitatively different pattern of performance. This is the first study contrasting input and output processes associated with phonological processing. The results suggest that deficits in SLI plus RDI may be associated with impairment in actively maintaining phonological representations for phonological processing, which is not present in those without RDI and which leads to reading decoding difficulties.

Psychometric properties of the Positive and Negative Affect Schedule (PANAS) in a heterogeneous sample of substance users

ABSTRACTBackground: The Positive and Negative Affect Schedule (PANAS) is a widely used measure of affect. A comprehensive psychometric evaluation among substance users, however, has not been published. Objective: To examine the psychometric properties of the PANAS in a sample of outpatient treatment substance users. Methods: We used pooled data from four randomized clinical trials (N = 416; 34% female, 48% African American). Results: A confirmatory factor analysis indicated adequate support for a two-factor correlated model comprised of Positive Affect and Negative Affect with correlated item errors (Comparative Fit Index = 0.93, Root Mean Square Error of Approximation = 0.07, χ2 = 478.93, df = 156). Cronbach’s α indicated excellent internal consistency for both factors (0.90 and 0.91, respectively). The PANAS factors had good convergence and discriminability (Composite Reliability > 0.7; Maximum Shared Variance < Average Variance Extracted). A comparison from baseline to Week 1 indicated acceptable test-retest reliability (Positive Affect = 0.80, Negative Affect = 0.76). Concurrent and discriminant validity were demonstrated with correlations with the Brief Symptom Inventory and Addiction Severity Index. The PANAS scores were also significantly correlated with treatment outcomes (e.g. Positive Affect was associated with the maximum days of consecutive abstinence from primary substance of abuse, r = 0.16, p = 0.001). Conclusion: Our data suggest that the psychometric properties of the PANAS are retained in substance using populations. Although several studies have focused on the role of Negative Affect, our findings suggest that Positive Affect may also be an important factor in substance use treatment outcomes.

Confirmatory factor analysis of VISA-P scale and measurement invariance across sexes in athletes with patellar tendinopathy

BackgroundThe Victorian Institute of Sport Assessment–Patella (VISA-P) scale is the most condition-specific patient-reported outcome measure used to assess symptom severity in athletes with patellar tendinopathy. Previous exploratory factor analyses have been conducted to evaluate the scale's dimensionality, with inconsistent results, and the factor structure of the scale remains unclear. The aims of the present study were to determine the factorial structure of the VISA-P scale using confirmatory factor analysis (CFA) and test measurement invariance across sexes. MethodsThe study included a convenience sample of 249 Spanish athletes with patellar tendinopathy. CFA was performed to assess factorial validity. Hypothesized 1- and 2-factor models were tested. Measurement invariance across sexes was evaluated via multi-group CFA with several fit indices using EQS 6.1 software. ResultsThe internal consistency coefficient was 0.74. Several CFA models were examined and the 1-factor model in which errors for Items 7 and 8 were correlated showed acceptable fit in terms of comparative fit index (CFI) and goodness-of-fit index (GFI) statistics (CFI = 0.93; GFI = 0.94; standardized root mean square residual = 0.06; root mean square error of approximation = 0.10; 90% confidence interval: 0.08–0.13). This model was invariant across sexes. ConclusionThe 1-factor model of the Spanish version of the VISA-P scale (VISA-P-Sp) in which errors for Items 7 and 8 were correlated demonstrated relative fit in CFA. Scores obtained via VISA-P-Sp can be compared between men and women without sexes bias. Further studies should examine the VISA-P scale and other single-score patient-reported outcome measures concurrently.

Errors in veterinary practice: preliminary lessons for building better veterinary teams

Case studies in two typical UK veterinary practices were undertaken to explore teamwork, including interprofessional working. Each study involved one week of whole team observation based on practice locations (reception,...

Age and gender differences in spatial perspective taking.

It is often necessary in daily experience to change one's point of view to adopt mentally the spatial perspective of other persons, learn the position of different objects in a new environment or even describe an environment to other persons. Hence, the ability to link spatial information from different perspectives seems to be necessary to orient ourselves in the space. Several studies have found gender-related differences in spatial reasoning in younger adults, but little is known about such effects in middle-aged and older adults. This research was designed to study how spatial perspective taking is affected by gender and age along the lifespan. The Perspective Taking/Spatial Orientation Test (PPT; Kozhevnikov and Hegarty [1]) was administered to groups of younger, middle-aged, and older adults, with females and males represented in each age group. The performance in the PPT decreased across age groups. All age groups had more errors in items that involved perspective changes of greater than 90º. Males performed better than females on most of the variables; however, no significant differences appeared in the interaction gender × age. The present findings showed the relevance of the degree perspective change in visuo-spatial abilities, especially in the older group. In relation with the gender, males outperformed females; however, the interaction gender×age did not show significant differences.

How do subvocal rehearsal and general attentional resources contribute to verbal short-term memory span?

Whether rehearsal has a causal role in verbal STM has been controversial in the literature. Recent theories of working memory emphasize a role of attentional resources, but leave unclear how they contribute to verbal STM. Two experiments (with 49 and 102 adult participants, respectively) followed up previous studies with children, aiming to clarify the contributions of attentional capacity and rehearsal to verbal STM. Word length and presentation modality were manipulated. Experiment 1 focused on order errors, Experiment 2 on predicting individual differences in span from attentional capacity and articulation rate. Structural equation modeling showed clearly a major role of attentional capacity as a predictor of verbal STM span; but was inconclusive on whether rehearsal efficiency is an additional cause or a consequence of verbal STM. The effects of word length and modality on STM were replicated; a significant interaction was also found, showing a larger modality effect for long than short words, which replicates a previous finding on children. Item errors occurred more often with long words and correlated negatively with articulation rate. This set of findings seems to point to a role of rehearsal in maintaining item information. The probability of order errors per position increased linearly with list length. A revised version of a neo-Piagetian model was fit to the data of Experiment 2. That model was based on two parameters: attentional capacity (independently measured) and a free parameter representing loss of partly-activated information. The model could partly account for the results, but underestimated STM performance of the participants with smaller attentional capacity. It is concluded that modeling of verbal STM should consider individual and developmental differences in attentional capacity, rehearsal rate, and (perhaps) order representation.

An analysis of error patterns in children's backward digit recall in noise.

The purpose of the study was to determine whether perceptual masking or cognitive processing accounts for a decline in working memory performance in the presence of competing speech. The types and patterns of errors made on the backward digit span in quiet and multitalker babble at -5 dB signal-to-noise ratio (SNR) were analyzed. The errors were classified into two categories: item (if digits that were not presented in a list were repeated) and order (if correct digits were repeated but in an incorrect order). Fifty five children with normal hearing were included. All the children were aged between 7 years and 10 years. Repeated measures of analysis of variance (RM-ANOVA) revealed the main effects for error type and digit span length. In terms of listening condition interaction it was found that the order errors occurred more frequently than item errors in the degraded listening condition compared to quiet. In addition, children had more difficulty recalling the correct order of intermediate items, supporting strong primacy and recency effects. Decline in children's working memory performance was not primarily related to perceptual difficulties alone. The majority of errors was related to the maintenance of sequential order information, which suggests that reduced performance in competing speech may result from increased cognitive processing demands in noise.

Self-esteem Mediates Associations of Physical Activity with Anxiety in College Women

Why physically active people report lower anxiety than those who are inactive is not well understood. This study examined whether physical self-concept and self-esteem would mediate associations of self-reported physical activity with anxiety disorder symptoms in young women, a population with elevated risk for developing an anxiety disorder. College women (N = 1036, mean ± SD = 19.7 ± 2.9 yr) completed a physical activity recall, the Psychiatric Diagnostic Screening Questionnaire, and the Physical Self-Description Questionnaire. Structural equation modeling was used to test hypotheses. Physical activity had inverse, indirect associations with symptoms of social phobia, generalized anxiety disorder, and obsessive-compulsive disorder that were expressed through its positive association with specific and global physical self-concept and self-esteem. The results were independent of similar relations with symptoms of major depressive disorder as well as the estimates of body fatness and use of psychotropic medications. These correlational findings provide initial evidence to warrant experimental efficacy trials of whether physical activity will reduce the risk of anxiety disorders in young women by positive influences on physical self-concept and self-esteem.

Beyond the Score: Clinical Evaluation Through Formal Psychological Assessment

Questionnaires generate numerical scores, but endorsing different sets of items could produce the same score despite reflecting qualitatively different configurations of clinical features. Formal psychological assessment (FPA) attempts to overcome this by identifying the clinical features entailed by observed response patterns. This study illustrates an application of FPA to the cleaning subscale of a questionnaire assessing obsessive–compulsive symptoms and DSM–IV–TR diagnostic criteria for obsessive–compulsive disorder. A deterministic model of the items–criteria relationships was constructed by mapping each item to each diagnostic criterion. The resulting model was tested on a large community sample (N = 4,412). Results indicate that the theoretical model has adequate fit; item error rates and probabilities for each of the criteria are examined. Clinically relevant examples of the items–criteria relationships are discussed. Possible applications of FPA to personality assessment are also discussed, including long multidimensional questionnaires and questionnaires that use subtle item content.

The Effects of Positively and Negatively Worded Items on the Factor Structure of the UCLA Loneliness Scale

The purpose of this study was to evaluate the factor structure of the University of California, Los Angeles (UCLA) Loneliness Scale and examine possible wording effects on a sample of 1,429 students from the United Arab Emirates University. Correlated traits-correlated uniqueness as well as correlated traits-correlated methods were used to examine the scale’s eight-factor models. The results of confirmatory factor analysis with maximum likelihood estimation indicated two models provided the best fit. The first, a one-dimensional model, assumes the correlation of the standard errors in the positive items. The second assumes a one-dimensional model and two method factors that correspond to the negative and positive items.

Construct Validity and Generalizability of Simulation-Based Objective Structured Clinical Examination Scenarios.

It is not known if construct-related validity (progression of scores with different levels of training) and generalizability of Objective Structured Clinical Examination (OSCE) scenarios previously used with non-US graduating anesthesiology residents translate to a US training program. We assessed for progression of scores with training for a validated high-stakes simulation-based anesthesiology examination. Fifty US anesthesiology residents in postgraduate years (PGYs) 2 to 4 were evaluated in operating room, trauma, and resuscitation scenarios developed for and used in a high-stakes Israeli Anesthesiology Board examination, requiring a score of 70% on the checklist for passing (including all critical items). The OSCE error rate was lower for PGY-4 than PGY-2 residents in each field, and for most scenarios within each field. The critical item error rate was significantly lower for PGY-4 than PGY-3 residents in operating room scenarios, and for PGY-4 than PGY-2 residents in resuscitation scenarios. The final pass rate was significantly higher for PGY-3 and PGY-4 than PGY-2 residents in operating room scenarios, and also was significantly higher for PGY-4 than PGY-2 residents overall. PGY-4 residents had a better error rate, total scenarios score, general evaluation score, critical items error rate, and final pass rate than PGY-2 residents. The comparable error rates, performance grades, and pass rates for US PGY-4 and non-US (Israeli) graduating (PGY-4 equivalent) residents, and the progression of scores among US residents with training level, demonstrate the construct-related validity and generalizability of these high-stakes OSCE scenarios.

The effect of the electronic transmission of prescriptions on dispensing errors and prescription enhancements made in English community pharmacies: a naturalistic stepped wedge study

ObjectivesTo compare prevalence and types of dispensing errors and pharmacists’ labelling enhancements, for prescriptions transmitted electronically versus paper prescriptions.DesignNaturalistic stepped wedge study.Setting15 English community pharmacies.InterventionElectronic transmission of prescriptions between prescriber...

A Two-Period Newsvendor Setting with Product Extension and Shifting Demand Concerns

The objective of this study is to optimize an overall expected profit in a two-period newsvendor setting by jointly determining these two items' order sizes and selling prices. The item in the second period is a product extension of the item in the first period. An inadequate inventory is intentionally placed in the first period in order to stimulate more sales in the second period. The first item's unsatisfied demand is assumed to be transferred to the second selling period with a certain shifting demand rate. The two items' stochastic demand is assumed to be in a linear-additive form consisting of a deterministic demand and an error demand that is exponentially distributed. Including the classical newsvendor model as a special case, we first propose a pertinent model that overshadows the classical newsvendor setting in terms of profit performances, and surprisingly find that the first item's error demand should be completely sacrificed when facing a high shifting demand rate. Afterwards, we modify the model to identify that our inadequate stock strategy is indeed practicable, and then extend the model to predetermine the two items' selling prices that ensure the existence of maximum of the expected profit. Possible triggered effects, such as an increasing primary demand, an increasing error demand and a cheaper purchasing cost in the second period, are also illustrated by examples.

A framework for economic demand forecast evaluation

Accurate demand forecasting is crucial for companies, since both overage and shortage of products can reduce company profits. Numerous statistical measures have been proposed to evaluate the accuracy of forecasts, but these measures tend to neglect the different economic impact of forecast errors for different items. Motivated by industry practitioners’ need to evaluate their forecasting processes and identify potentials for improvement, we develop a framework to evaluate demand forecasts. The framework consists of statistical measures and a scoring model and allows for an analysis of forecast accuracy on different levels of aggregation. A case study from a large retailing company shows that the framework is easy to implement in standard software and can provide relevant managerial insights.