Distribution Of Test Scores Research Articles

Discreteness Causes Bias in Percentage-Based Comparisons: A Case Study From Educational Testing

Discretizing continuous distributions can lead to bias in parameter estimates. We present a case study from educational testing that illustrates dramatic consequences of discreteness when discretizing partitions differ across distributions. The percentage of test takers who score above a certain cutoff score (percent above cutoff, or “PAC”) often describes overall performance on a test. Year-over-year changes in PAC, or ΔPAC, have gained prominence under recent U.S. education policies, with public schools facing sanctions if they fail to meet PAC targets. In this article, we describe how test score distributions act as continuous distributions that are discretized inconsistently over time. We show that this can propagate considerable bias to PAC trends, where positive ΔPACs appear negative, and vice versa, for a substantial number of actual tests. A simple model shows that this bias applies to any comparison of PAC statistics in which values for one distribution are discretized differently from values for the other.

Can money ‘buy’ schooling achievement? Evidence from 19 Chinese cities

This paper examines the causal effect of private tutoring on Chinese and mathematics test scores of primary school students in urban China. Because the unobserved determinants of schooling achievement often also influence private tutoring expenditure, the OLS estimate cannot provide a consistent estimate. This paper adopts a heteroskedasticity-based identification strategy proposed by Lewbel (2012) to handle this problem. These estimation results show that, on average, private tutoring expenditure has small but statistically significant effect on the mathematics test score of primary school students, but has no statistically significant effect on the Chinese test score. A 1000 yuan increase in private tutoring expenditure (i.e. 54% of a standard deviation) raises the primary school students' mathematics test score by 1.07 percentage point (i.e. 15% of a standard deviation). The instrumental variable quantile regression combining with Lewbel IV suggests that private tutoring is more likely to improve student achievement at the bottom end of test score distribution. When moving upward to the top end, the effect becomes smaller and even negative although not significant.

The Effect of M@tabel on Italian Students’ Performances: A Quantile Regression Approach

The effectiveness of a training program on students’ achievements has primarily relied on estimation approaches which capture the mean effect on students’ performances. While estimating how “on average” variables affect educational outcomes yields straightforward interpretations, the standard methodology may miss what is crucial for policy purposes, namely how educational programs affect students achievements differently at different points of the conditional test score distribution. The aim of this study is to investigate the short term effects of M@tabel on Italian sixth grade students performance in mathematics at secondary schools through a multilevel quantile regression approach. The proposed model allows to fully characterize the entire conditional distribution of performances in mathematics, providing a more complete view of a possible relationship between M@tabel treatment and the observed math score gain. The main concern is not only “does the training program have an impact” but also the question “for whom does it matter, and how”.

Dynamics of the black–white gap in academic achievement

The black–white test score gap remains a measurable phenomenon in the United States. Up to this point the literature has primarily focused on the black–white achievement gap without taking into account the underlying mobility patterns of individual students as they progress from one grade to the next. However, the degree to which policy makers and educators should be concerned about the black–white test score gap should be tied to how mobile the two groups of students are through the distribution of test scores from one grade to the next. In this paper I apply two nonparametric estimators of distributional mobility to data on test scores and track black–white differences in mobility across the entire distribution of achievement. When compared to whites, blacks tend to be less upwardly mobile and more downwardly mobile for both math and reading. This pattern is particularly prominent for reading in the very early years of schooling.

Kernel Equating Under the Non-Equivalent Groups With Covariates Design.

When equating two tests, the traditional approach is to use common test takers and/or common items. Here, the idea is to use variables correlated with the test scores (e.g., school grades and other test scores) as a substitute for common items in a non-equivalent groups with covariates (NEC) design. This is performed in the framework of kernel equating and with an extension of the method developed for post-stratification equating in the non-equivalent groups with anchor test design. Real data from a college admissions test were used to illustrate the use of the design. The equated scores from the NEC design were compared with equated scores from the equivalent group (EG) design, that is, equating with no covariates as well as with equated scores when a constructed anchor test was used. The results indicate that the NEC design can produce lower standard errors compared with an EG design. When covariates were used together with an anchor test, the smallest standard errors were obtained over a large range of test scores. The results obtained, that an EG design equating can be improved by adjusting for differences in test score distributions caused by differences in the distribution of covariates, are useful in practice because not all standardized tests have anchor tests.

School context and gender differences in mathematical performance among school graduates in Russia

Gender differences in mathematical performance have received considerable scrutiny in the fields of sociology, economics and psychology. We analyse a large data-set of high school graduates who took a standardised mathematical test in Russia in 2011 (n = 738,456) and find no substantial difference in mean test scores across boys and girls. However, boys have a greater variance of scores and are more numerous at the top of the distribution. We apply quantile regression to model the association between school characteristics and gender differences in test scores throughout the distribution of test scores. Male advantage in test scores, particularly at the top of the distribution, is concentrated in cities and in the schools with an advanced curriculum. In other high schools, especially in the countryside, gender differences in all parts of the distribution are small. We suggest several mechanisms based on selection and school effects that account for our findings.

Grades and rank: Impacts of non-financial incentives on test performance

How does effort respond to being graded and ranked? This paper examines the effects of non-financial incentives on test performance. We conduct a randomized field experiment on more than a thousand sixth graders in Swedish primary schools. Extrinsic non-financial incentives play an important role in motivating highly skilled students to exert more effort. We find significant differences in test scores between the intrinsically motivated control group and three of four extrinsically motivated treatment groups. The only treatment not increasing test performance is criterion-based grading on an A–F scale, which is the typical grading method. Test performance is significantly higher if employing rank-based grading or giving students a symbolic reward. The motivational strengths of the non-financial incentives differ across the test score distribution, across the skill distribution, with peer familiarity, and with respect to gender. Boys are only motivated by rank-based incentives, while girls are also motivated by receiving a symbolic reward. Rank-based grading and symbolic rewards tend to crowd out intrinsic motivation for students with low skills, while girls also respond less to rank-based incentives if tested with less familiar peers.

The impact of maternal depression on children’s cognitive development: An analysis based on panel quantile regressions

We use unconditional quantile regressions for panel data to capture the various effects of maternal depression along the actual distribution of children’s cognitive test scores. Children falling in the middle of the test score distribution are most impacted.

Semiparametric decomposition of the gender achievement gap: An application for Turkey

This study employs a semiparametric Oaxaca–Blinder (OB) decomposition to investigate the gender PISA test score gap in mathematics/science in Turkey. This technique, which has not previously appeared in the gender achievement gap literature, relaxes the parametric assumptions of the standard OB decomposition, accounts for the possible violation of the common support assumption, and allows us to explore the gender test score gap not only at the mean but also across the test score distribution. The results from the semiparametric OB decomposition of the mean test score gap indicate that girls outperform boys in science whereas the gap is not statistically significant in mathematics. We also find that the mean gap fails to uncover the heterogeneous pattern that the gap exhibits across the distribution.

Descriptive Statistics for Modern Test Score Distributions: Skewness, Kurtosis, Discreteness, and Ceiling Effects.

Many statistical analyses benefit from the assumption that unconditional or conditional distributions are continuous and normal. More than 50 years ago in this journal, Lord and Cook chronicled departures from normality in educational tests, and Micerri similarly showed that the normality assumption is met rarely in educational and psychological practice. In this article, the authors extend these previous analyses to state-level educational test score distributions that are an increasingly common target of high-stakes analysis and interpretation. Among 504 scale-score and raw-score distributions from state testing programs from recent years, nonnormal distributions are common and are often associated with particular state programs. The authors explain how scaling procedures from item response theory lead to nonnormal distributions as well as unusual patterns of discreteness. The authors recommend that distributional descriptive statistics be calculated routinely to inform model selection for large-scale test score data, and they illustrate consequences of nonnormality using sensitivity studies that compare baseline results to those from normalized score scales.

Within-person distributions of neuropsychological test scores as a function of dementia severity.

The distribution of a person's scores on a battery of neuropsychological tests reflects the nature and extent of intraindividual variability and may provide clinically useful information not captured by examining test scores in isolation. We sought to test the hypothesis that systematic alterations of within-person test-score distributions characterize worsening cognitive impairment. We analyzed cross-sectional data from 2 datasets that included 395 clinical patients and 135 neurologically normal older adults (≥60 years old). We computed each person's mean, standard deviation, skewness, and kurtosis for a battery of 13 neuropsychological measures and compared these distributional properties across groups. Most healthy older adults produced normal (Gaussian) test-score distributions. Among patients, test-score distributions increasingly shifted away from normal with worsening cognitive impairment. Worsening dementia was accompanied by progressively lower mean scores and increasingly positive skew. Within-person standard deviations initially grew at mild levels of impairment, but then shrank with worsening dementia, resulting in positive kurtosis. Within-person distributional properties vary as a function of dementia severity. Despite the limitations associated with using a clinical rather than a research sample, the analyses reported here serve as a "proof of concept" and suggest that similar investigations with more rigorously characterized patient samples may be fruitful.

"Within-person distributions of neuropsychological test scores as a function of dementia severity": Correction to Reckess, Varvaris, Gordon, and Schretlen (2013).

"Within-person distributions of neuropsychological test scores as a function of dementia severity": Correction to Reckess, Varvaris, Gordon, and Schretlen (2013).

Music Perception in Older Adult Cochlear Implant Users: Evidence About Age-related Decline in Rapid Temporal Processing

Age-related cognitive decline (ARCD) refers to deterioration of basic cognitive functions in normal human aging that mainly associates with areas of sensory processing, attention, memory, and executive control. Because listening to music involves a highly complex integration in various levels of cognitive processing that extend from basic auditrory fuctions to high-level feature extraction and emotional/aesthetic response, music listening tests have been frequently used in clinical reserch settings in the past two decades as a tool for screening areas of deficits in various kinds of pathological populations (e.g., hearing impaired, neurological patients, children with learning disabilities). There are two main reasons why assessing music listening abilities of older adult Cochlear Implant (CI) users is important for aging researchers. First, it shows how the combined influence of ARCD, and the reduced and/or distorted sensory information provided by a hearing implant may globally or selectively deteriorate an implantee's sensitivity to basic elements of music such as rhythm, melody, harmonic structure, and consonance/dissonance. Second, it provides evidence about cognitive domains or processes that display the greatest decline with age.In this study the effects of CI users’ chronological age on their performance on music listening tests were explored through the use of the MuSIC perception test battery, with special emphasis on the influence of aging in perceiving musical stimuli that require rapid temporal processing (e.g., rhythm patterns). Analysis is based on data from a recently completed European multicenter study on music perception in CI users wearing a MED-EL's PULSARCI100 or a SONATATI100 with an OPUS 2 audio processor (N = 81, age range: 21,2-82,3 years), and Normally Hearing (NH) adults (N = 46, age range: 21,1-58,8 years). Areas of music listening abilities that display noticeable declines in older implant users are highlighted by statistically comparing distributions of test scores among a subgroup of CI users over 60 years (N = 24), the subgroup of users younger than 60, and the group of NH controls. Resultsshowed that older adult CI users appear not to differ significantly from younger ones in processing low-level features of music such as pitch and timbre, although their accuracy is differentially affected in favor of the younger ones when listening tasks are complex and require rapid temporal processing of the stimuli, or attention switching between two or more sources of information (e.g., identification of individual musical instruments playing in ensemble). Aging also appears to negatively affect processing of rhythm patterns in music, a deficit likely attributable to general slowing of information processing in older adults, while, in contrast, processing of melodies and chord structures remains relatively intact.

Cross-Validation

The development of the kernel equating (KE) method enhanced the theory of observed-score equating. In KE, discrete test score distributions are converted into continuous distributions through the use of a Gaussian kernel. Traditionally, the optimal bandwidth for a Gaussian kernel was obtained by minimizing a penalty function. In this article, an alternative bandwidth-selection approach for KE was adopted that uses cross-validation (CV) techniques. The method is illustrated through simulations that were conducted with 188 conditions by varying three factors known to influence equating results; these include sample sizes, score distributions, and methods that involve both equating and bandwidth-selection methods. Four equating procedures were considered: traditional equipercentile equating, which uses linear interpolation to make the test distributions continuous; KE with penalty functions; and KE with two newly proposed CV methods. The results were evaluated based on four criteria: bias in continuizing the distributions (i.e., the difference between the estimated and underlying score distributions), the standard error of equating (SEE), the difference between equated scores, and percent relative error (PRE). Overall, the results demonstrate that KE with the two CV methods outperformed the others—the estimated density functions were less biased and the SEEs and PREs were smaller. Equating differences between the different methods were produced, although they were not large. In addition, the bias issues surrounding kernel methods on sample sizes and the shapes of the distributions were addressed and discussed.

Grades and Rank: Impacts of Non-Financial Incentives on Test Performance

How does effort respond to being graded and ranked? This paper examines the effects of non-financial incentives on test performance. We conduct a randomized field experiment on more than a thousand sixth graders in Swedish primary schools. Extrinsic non-financial incentives play an important role in motivating highly skilled students to exert more effort. We find significant differences in test scores between the intrinsically motivated control group and three of four extrinsically motivated treatment groups. The only treatment not increasing test performance is criterion-based grading on an A-F scale, which is the typical grading method. Test performance is significantly higher if employing rank-based grading or giving students a symbolic reward. The motivational strengths of the non- financial incentives differ across the test score distribution, across the skill distribution, with peer familiarity, and with respect to gender. Boys are only motivated by rank-based incentives, while girls are also motivated by receiving a symbolic reward. Rank-based grading and symbolic rewards tend to crowd out intrinsic motivation for students with low skills, while girls also respond less to rank-based incentives if tested with less familiar peers.

Analyzing Educational Achievement Differences between Second-Generation Immigrants: Comparing Germany and German-Speaking Switzerland

In this study, I provide evidence that the educational achievement of second-generation immigrants in German-speaking Switzerland is greater than in Germany. The impact of the first-generation immigrants' destination decision on their offspring's educational achievement seems to be much more important than has been recognized by the existing literature. I identify the test score gap between these students that cannot be explained by differences in individual and family characteristics. Moreover, I show how this gap evolves over the test score distribution and how the least favorably-endowed students fare. My results suggest that the educational system of Switzerland, relative to the German system, enhances the performance of immigrants' children substantially. This disparity is largest when conditioning on the language spoken at home, and prevails even when comparing only students whose parents migrated from the same country of origin.

Grades and Rank: Impacts of Non-Financial Incentives on Test Performance

How does effort respond to being graded and ranked? This paper examines the effects of non-financial incentives on test performance. We conduct a randomized field experiment on more than a thousand sixth graders in Swedish primary schools. Extrinsic non-financial incentives play an important role in motivating highly skilled students to exert more effort. We find significant differences in test scores between the intrinsically motivated control group and three of four extrinsically motivated treatment groups. The only treatment not increasing test performance is criterion-based grading on an A-F scale, which is the typical grading method. Test performance is significantly higher if employing rank-based grading or giving students a symbolic reward. The motivational strengths of the non-financial incentives differ across the test score distribution, across the skill distribution, with peer familiarity, and with respect to gender. Boys are only motivated by rank-based incentives, while girls are also motivated by receiving a symbolic reward. Rank-based grading and symbolic rewards tend to crowd out intrinsic motivation for students with low skills, while girls also respond less to rank-based incentives if tested with less familiar peers.

Spillover Effects of Studying with Immigrant Students; A Quantile Regression Approach

We analyze how the share of immigrant children in the classroom affects the educational attainment of native Dutch children in terms of their language and math performance at the end of primary school. Our paper studies the spill-over effects at different parts of the test score distribution of native Dutch students using a quantile regression approach. We find no evidence of negative spillover effects of the classroom presence of immigrant children at the median of the test score distribution. In addition, there is no indication that these spill-over effects are present at other parts of the distribution.

Two Distinct Amnesic Profiles in Behavioral Variant Frontotemporal Dementia

Whether or not episodic memory deficit is a characteristic of behavioral variant frontotemporal dementia (bvFTD) is a crucial question for its diagnosis and management. We compared the episodic memory performance profile of bvFTD patients with healthy control subjects and patients with Alzheimer's disease (AD) as defined by clinical and biological criteria. Episodic memory was assessed with the Free and Cued Selective Reminding Test, which controls for effective encoding and identifies memory storage ability resulting from consolidation processing. One hundred thirty-four participants were evaluated: 56 patients with typical clinical presentation of AD and pathophysiological evidence as defined by cerebrospinal fluid AD biomarker profile and/or significant amyloid retention on Pittsburgh Compound B positron emission tomography; 56 patients diagnosed with bvFTD with no evidence of AD-cerebrospinal fluid biomarkers when a profile was available (28/56), including 44 progressive (bvFTD) and 12 nonprogressive (phenocopies) patients; and 22 control subjects with negative amyloid imaging. Memory scores could not differentiate bvFTD from AD patients (sensitivity and specificity <50%). Taking into account the individual distribution of Free and Cued Selective Reminding Test scores, half of bvFTD patients had a deficit of free recall, total (free + cued) recall, and delayed recall as severe as AD patients. The other half had subnormal scores similar to phenocopies and a delayed recall score similar to control subjects. We observed two distinct amnesic profiles in bvFTD patients that could reflect two types of hippocampal structure and Papez circuit involvement. These findings on episodic memory profiles could contribute to discussions on the recent international consensus criteria for bvFTD.

Survey results regarding use of the Boston Naming Test: Houston, we have a problem

Members of the National Academy of Neuropsychology were surveyed in 2005 to assess then current practices regarding Boston Naming Test (BNT) administration, interpretation, and reporting procedures. Nearly half of 445 respondents followed discontinuation rules that differed from instructions published with the test, and nearly 10% did not administer items in reverse order to achieve the required 8 consecutive item basal. Of further concern, between 40% and 55% of respondents indicated that they did not interpret BNT scores in light of linguistic and ethnic background, and over 25% reported that they did not consider educational level. Despite the fact that non-normal distribution of BNT test scores renders use of percentiles misleading, nearly 60% of respondents endorsed using percentiles when reporting BNT data. The implications of these results are discussed, and recommendations are provided.