Difficult Test Items Research Articles

Setting Standards With Multiple-Choice Tests: A Preliminary Intended-User Evaluation of SmartStandardSet

Software that easily helps higher education instructors to remove poor quality items and set appropriate grade boundaries is generally lacking. To address these challenges, the SmartStandardSet system provides a graphical-user interface for removing defective items, weighting student scores using a two-parameter model IRT score analysis, and a mechanism for standard-setting. We evaluated the system through a series of six interviews with teachers and six focus groups involving 19 students to understand how key stakeholders would view the use of the tool in practice. Generally, both groups of participants reported high levels of feasibility, accuracy, and utility in SmartStandardSet’s statistical scoring of items and score calculation for test-takers. Teachers indicated the data displays would help them improve future test items; students indicated the system would be fairer and would motivate greater effort on more difficult test items. However, both groups had concerns about implementing the system without institutional policy endorsement. Students specifically were concerned that academics may set grade boundaries on arbitrary and invalid grounds. Our results provide useful insights into the perceived benefits of using the tool for standard setting, and suggest concrete next steps for gaining wider acceptance that will be the focus of future work.

The effects of theta transcranial alternating current stimulation (tACS) on fluid intelligence

The objective of the study was to explore the influence of transcranial alternating current stimulation (tACS) on resting brain activity and on measures of fluid intelligence. Theta tACS was applied to the left parietal and left frontal brain areas of healthy participants after which resting electroencephalogram (EEG) data was recorded. Following sham/active stimulation, the participants solved two tests of fluid intelligence while their EEG was recorded. The results showed that active theta tACS affected spectral power in theta and alpha frequency bands. In addition, active theta tACS improved performance on tests of fluid intelligence. This influence was more pronounced in the group of participants that received stimulation to the left parietal area than in the group of participants that received stimulation to the left frontal area. Left parietal tACS increased performance on the difficult test items of both tests (RAPM and PF&C) whereas left frontal tACS increased performance only on the easy test items of one test (RAPM). The observed behavioral tACS influences were also accompanied by changes in neuroelectric activity. The behavioral and neuroelectric data tentatively support the P-FIT neurobiological model of intelligence.

Study on Risk Identification and Prevention of Power Transformer Based on Sampling Inspection Theory

Power transformer is one of the key equipment for power grid, and its quality has important effects to the security, stability and economic operation of the power grid. The government sampling inspection pass rate of 10kV power transformer is less than 80% in China, which causes serious impact to the product quality and brings serious security risks. It has great significance to study on the risk identification and prevention method. In this paper, the sampling inspection theoretical analysis of power transformer is carried out with the mathematical statistics and probability theory. Its application in the quality risk identification is studied on. The risk prevention effect of the sampling inspection is studied based on game theory. It is used both on the large scale power transformer and distribution transformer. The sampling inspection has the characteristics such as flexibility, randomness, science, and economics. It can be used in large range with flexible way, especially for the difficult test items. Random is the foundation for the scientific sampling inspection. The approach determines that it can reflect the population quality level with small cost, which has obvious economic benefits. In practice, sampling inspection should focus on the key aspects, including the design of sampling inspection plan, the selection of the sampling mode, the cost and the corresponding punishment.

Major Reforms and Issues of the Medical Licensing Examination Systems in Korea

Since the establishment of the national medical licensing examination board in 1992, the medical licensing examination system has changed enormously and this has had a number of impacts on examination services. All those reforms were aimed at improving the relevance and reliability of the test. Several attempts of the testing system have appeared in the new examination service, and which have also brought about the changes in the medical school curriculum such as introducing integrated courses instead of traditional subjects, using test scores as a reference to the post-graduation selection test. Some examples of changes in the examination system are as follows: 1) choosing three integrated test subjects and outlines of their reference content instead of 15 academic subjects, 2) adjusting the ratio of multiple choice question items to focus more on the problem solving level, 3) introduction of ‘one-best answer’ single set and ‘extended matching type items, 4) item construction based on real clinical cases and real clinical materials. Recently, a clinical skill test system has been introduced to measure examinees’ basic clinical skills competencies. Despite continuing efforts, the examination system still has many issues remaining to be solved. These problems include the differential weighting of test items, appropriate threshold for passing, and practicality of pre-testing to stabilize the passing rate and avoid the hazards of newness and undesirably difficult test items.

Analysis of the Test Content and Characteristics of the Highly Difficult Test Items and the Attractive Distractors within the Techniques in Basic Agricultural Subject of the CSAT

Analysis of the Test Content and Characteristics of the Highly Difficult Test Items and the Attractive Distractors within the Techniques in Basic Agricultural Subject of the CSAT

“Don't Call Me a Student-Athlete”: The Effect of Identity Priming on Stereotype Threat for Academically Engaged African American College Athletes

Academically engaged African American college athletes are most susceptible to stereotype threat in the classroom when the context links their unique status as both scholar and athlete. After completing a measure of academic engagement, African American and White college athletes completed a test of verbal reasoning. To vary stereotype threat, they first indicated their status as a scholar-athlete, an athlete, or as a research participant on the cover page. Compared to the other groups, academically engaged African American college athletes performed poorly on the difficult test items when primed for their athletic identity, but they performed worse on both the difficult and easy test items when primed for their identity as a scholar-athlete. The unique stereotype threat processes that affect the academic performance of minority college athletes are discussed.

The Role of Gender Identities and Stereotype Salience With the Academic Performance of Male and Female College Athletes

An experiment was conducted to examine factors that moderate the experience of academic identity threat among college athletes who represent a stigmatized group on most college campuses (Yopyk & Prentice, 2005). It was hypothesized that because they are more engaged in academics, female college athletes would be especially threatened by the prospect of confirming the “dumb-jock” stereotype. As predicted, female college athletes performed more poorly when their athletic and academic identities were explicitly linked, but only on moderately difficult test items. The results also revealed that male college athletes performed significantly better (see stereotype reactance and self-affirmation) on more difficult test items when only their athletic identity was primed prior to the test. This is an important finding as there is little research on the impact of positive stereotypes on performance. The discussion focuses on the different motivational processes (i.e. self-affirmation) that impact the academic performance of male and female college athletes when aspects of their campus identity are primed within a classroom context.

Onderzoek Naar Oorzaken van 'Differential Item Functioning' in de Eindtoets Basisonderwijs

Previous research has shown that the Dutch 'Final Test of Primary Education' contains a number of unintentionally and therefore unwanted, difficult test items, leading to Differential Item Functioning (DIF) for immigrant minority students whose parents' dominant language is Turkish or Arab/Berber. Two statistical procedures were used to identify DIF-items in the Final Test of 1997. Subsequently, five experiments were conducted to detect causes of DIF, revealing a number of hypotheses concerning possible linguistic, cultural, and textual sources. These hypotheses were used to manipulate original DIF-items into intentionally DIF-free items. The article discusses three possible sources of DIF: (1) the use of fixed (misleading) answer-options and (2) of misleading illustrations (both in the disadvantage of the minority students), and (3) the fact that questions concerning past tense often lead to DIF (in their advantage).

Achievement Results for Second and Third Graders Using the Standards-Based Curriculum Everyday Mathematics

Students using Everyday Mathematics (EM), developed to incorporate ideas from the NCTM Standards, were at normative U.S. levels on multidigit addition and subtraction symbolic computation on traditional, reform-based, and EM-specific test items. Heterogeneous EM 2nd graders scored higher than middle- to upper-middle-class U.S. traditional students on 2 number sense items, matched them on others, and were equivalent to a middle-class Japanese group. On a computation test, the EM 2nd graders outperformed the U.S. traditional students on 3 items involving 3-digit numbers and were outperformed on the 6 most difficult test items by the Japanese children. EM 3rd graders outscored traditional U.S. students on place value and numeration, reasoning, geometry, data, and number-story items.

The performance effects of headings within multiple‐choice tests

Background. It has been suggested that headings within multiple‐choice tests positively affect test performance; however, there is little research to date to either support or reject this hypothesis.Aims. Do headings within multiple‐choice tests facilitate test performance?Sample. The available sample for this study included employees of a large manufacturing plant in the western part of the United States (N=143). Participants ranged in age from 26 to 64 years old; the median age was 44 years old.Method. The experimental group received a multiple‐choice recertification test with headings included; the control group received the same test without headings. The data were analysed by analysis of covariance.Results. A significant difference was obtained for the treatment main effect, F (1,140) =4.33 (p<.04), with a calculated effect size for headings over no headings of es=0.33. Post hoc analysis of item difficulty data suggests that headings may have the greatest effect on less difficult test items and have little effect on difficult test items.Conclusions. The findings of the present study indicate that headings within multiple‐choice tests positively affect test performance. Perhaps more importantly, the findings indicate that headings may differentially impact test items of differing difficulty. Further research is recommended.

Text Processing Variables Predict the Readability of Everyday Documents Read by Older Adults

A model is presented to predict the readability of documents encountered by older adults. The documents studied are contained in the Educational Testing Service's Test of Basic Skills (1977 edition) and require readers to answer questions about charts (e.g., bus schedules), labels (e.g., plant spray labels and prescriptions), and forms (e.g., tax forms). The components of the model came from theoretical and empirical work on discourse processing and include such factors as discourse structure, emphasis, and position of an answer in a linguistic analysis of the everyday document.A sample of 482 adults from 52 to 93 years of age took the everyday problems test as well as a psychometric ability battery. The correlation was .54 (p < .01) between the readability scores for test items predicted by the model and the percentage of older adults correctly answering those items. In addition, the more difficult test items as identified by the model were correlated more highly with fluid intelligence abilities (figural relations and induction), crystallized intelligence abilities (vocabulary, experiential evaluation), and with memory span.

Hypnosis, Suggestion Type, and Subjective Experience — The Order-Effects Hypothesis Revisited:A Brief Communication

In a replication and extension of Field, Evans, and Orne's (1965) research, no support was found for the hypothesis that suggestion order is related to hypnotic responding. Confirming earlier findings, Ss were no more responsive to suggestions ordered from easy-to-difficult than they were to suggestions ordered from difficult-to-easy. Measures of subjective involvement in suggestions, involuntariness, and archaic involvement with the hypnotist were no more sensitive to order effects than was the measure of objective responding. Nor were order effects more apparent with Ss who received direct versus indirect suggestions. Direct suggestions facilitated suggestion-related involuntariness and response to the hypnotic amnesia item after cancellation, whereas indirect suggestions enhanced fears of negative appraisal by the hypnotist. Finally, female Ss were more involved in suggestions than were the males, particularly in response to more difficult test items.

Reserve capacity of the elderly in aging-sensitive tests of fluid intelligence: replication and extension.

Fluid intelligence belongs to that cluster of intellectual abilities evincing aging loss. To examine further the range of intellectual reserve available to aging individuals and the question of replicability in a new cultural and laboratory setting, 204 healthy older adults (mean age = 72 years; range = 60-86) participated in a short-term longitudinal training study. For experimental subjects, 10 sessions consisted of cognitive training involving two subability tests (Figural Relations, Induction) of fluid intelligence. The pattern of outcomes replicates and expands on earlier studies. Older adults have the reserve to evince substantial increases in levels of performance in fluid intelligence tests. Transfer of training, however, is narrow in scope. Training also increases accuracy of performance and the ability to solve more difficult test items. Difficulty level was estimated in a separate study, with a comparable sample of N = 112 elderly adults. Future research is suggested to examine whether intellectual reserve extends to near-maximum levels of performance.

Reserve capacity of the elderly in aging-sensitive tests of fluid intelligence: Replication and extension.

Baltes, P.B., Dittmann-Kohli, F. & Kliegl, R. (1986). Reserve capacity of the elderly in aging-sensitive tests of fluid intelligence: Replication and extension. Psychology and Aging , 1 , 172-177. Fluid intelligence belongs to that cluster of intellectual abilities evincing aging loss. To examine further the range of intellectual reserve available to aging individuals and the question of replicability in a new cultural and laboratory setting, 204 healthy older adults (mean age = 72 years; range = 60-86) participated in a short-term longitudinal training study. For experimental subjects, 10 sessions consisted of cognitive training involving two subability tests (Figural Relations, Induction) of fluid intelligence. The pattern of outcomes replicates and expands on earlier studies. Older adults have the reserve to evince substantial increases in levels of performance in fluid intelligence tests. Transfer of training, however, is narrow in scope. Training also increases accuracy of performance and the ability to solve more difficult test items. Difficulty level was estimated in a separate study, with a comparable sample of N = 112 elderly adults. Future research is suggested to examine whether intellectual reserve extends to near-maximum levels of performance. DOI 10.1037/0882-7974.1.2.172 Psychology and Aging

Reading retardation or linguistic deficit? II: test‐answering strategies in hearing and hearing‐impaired school children

ABSTRACTThe reading test performances of 60 hearing and 60 hearing‐impaired children of similar measured reading ages on the Southgate reading test were analysed. As in an earlier study using the Brimer Wide‐span test it was shown that the performances of the two groups were quite different. Deaf children tackled significantly more test items than the hearing and made significantly more errors in achieving similar reading scores.A detailed examination of both correct and incorrect answers showed that the deaf children were not simply providing answers to questions at random. Even where they produced incorrect responses they tended, as a group, to select the same answer. Unlike the hearing group, who did not converge on the same incorrect solution to difficult test items, the deaf were systematic in their choices, indicating that they were using a consistent strategy.A post hoc examination of individual test items indicated that the deaf children were selecting answers on the basis of word associations in each test item. On some items these produced a correct response, on others the same (incorrect) response. The implications of these findings are discussed to argue that reading tests based on hearing norms are of little value in the assessment of reading abilities and reading problems in hearing‐impaired children.