Music Performance Assessment Research Articles

Evaluating Differential Rater Functioning Over Time in the Context of Solo Music Performance Assessment

Abstract Rater variability studies in the context of music performance assessment treat rater effects as static characteristics of raters, where the effects occur similarly across each assessed performance. The purpose of this study was to investigate expert raters’ (N = 13) differential severity/leniency as dynamic processes, where the rater effects occur over time. In particular, we sought to examine the manifestation of group and individual variability using a class of rater effects referred to as differential rater functioning over time (DRIFT). DRIFT refers to the changes in rater performance in relation to a parameter of time. Three classes of Multifaceted Rasch (MFR) models were specified in order to explore differences in raters’ systematic changes in their interpretation of a 4-point rating scale structure across a 5-day rating session: (a) time-static model, (b) rater-by-time interaction model, and (c) partial credit model for time points. Results indicated a significant difference in severity/leniency across time for both the group of raters as a whole and some individual raters. Overall, raters demonstrated a general trend of decreasing severity over the 5-day rating session. Interaction analyses suggested that differential severity/leniency existed for both the raters as a group and for 9 out of the 13 individual raters. Of the total 65 potential pairwise interaction terms examined between raters and days, 21 (33.31%) were found to be statistically significant. Ten interactions systematically underestimated the performances and 11 interactions systematically overestimated the performances. Implications for the improved fairness of ratings in music assessment contexts are discussed.

Assessment of instrumental music performance: definitions, criteria, measurement

Assessment of instrumental music performance: definitions, criteria, measurement

Exploring the Effects of Rater Linking Designs and Rater Fit on Achievement Estimates Within the Context of Music Performance Assessments

When good model-data fit is observed, the Many-Facet Rasch (MFR) model acts as a linking and equating model that can be used to estimate student achievement, item difficulties, and rater severity on the same linear continuum. Given sufficient connectivity among the facets, the MFR model provides estimates of student achievement that are equated to control for differences in rater severity. Although several different linking designs are used in practice to establish connectivity, the implications of design differences have not been fully explored. Research is also limited related to the impact of model-data fit on the quality of MFR model-based adjustments for rater severity. This study explores the effects of linking designs and model-data fit for raters on the interpretation of student achievement estimates within the context of performance assessments in music. Results indicate that performances cannot be effectively adjusted for rater effects when inadequate linking or model-data fit is present.

Exploring rater cognition: A typology of raters in the context of music performance assessment

This manuscript sought to investigate rater cognition by exploring rater types based upon differential severity and leniency associated with rating scale items, rating scale category functioning, and dimensions of music performance assessment. The purpose of this study was to empirically identify typologies of operational raters based upon systematic differential severity indices in the context of large ensemble music performance assessment. A rater cognition information-processing model was explored based upon two frameworks: a framework for scoring and a framework for audition. Rater scoring behavior was examined using a framework for scoring, where raters’ mental processes compare auditory images to the scoring criteria used to generate a scoring decision. The scoring decisions were evaluated using the Multifaceted Rasch Partial Credit Measurement Model. A rater typology was then examined under the framework of audition, where similar schemata were defined through raters’ clustering of differential severity indices related to items and compared across performance dimensions. The results provided three distinct rater-types: (a) the syntactical rater; (b) the expressive rater; and (c) the mental representation rater. Implications for fairness and precision in the assessment process are discussed as well as considerations for validity of scoring processes.

What you see is what you hear: The importance of visual priming in music performer identification

Visual information plays a critical role in the assessment of music performance. Audiovisual integration is well recognised in person perception, and people readily match talking faces to speaking voices. This effect exists in identifying music performers, but its strength is untested. This study investigated the importance of visual or audio priming in identifying a music performer from a line-up. Half the participants saw a target saxophonist (no sound) and then heard a line-up (no visuals) of saxophonists playing (2 to 5 saxophonists). In contrast, half the participants heard a target saxophonist (no visuals) and then saw the line-up (no sound). Participants identified the target saxophonist in visual and audio line-ups at a rate above chance, although identification accuracy decreased as the line-up number increased. Those who saw the targets identified a significantly greater number of performers from the audio line-up than those who heard the targets and identified them from a visual line-up. As the task complexity and number of distractors increased, responses remained consistent and visual priming was robust and reliable in performer identification.

Examining Rater Precision in Music Performance Assessment

The use of raters as a methodological tool to detect significant differences in performances and as a means to evaluate music performance achievement is a solidly defended practice in music psychology, education, and performance science research. However, psychometric concerns exist in raters’ precision in the use of task-specific scoring criteria. A methodology for managing rater quality in rater-mediated assessment practices has not been systematically developed in the field of music. The purpose of this study was to examine rater precision through the analysis of rating scale category structure across a set of raters and items within the context of large-group music performance assessment using a Multifaceted Rasch Partial Credit (MFR-PC) Measurement Model. Allowing for separate parameterization estimation of the rating scale for each rater can more clearly detect variability in rater judgment and improve model-data fit, thereby enhancing objectivity, fairness, and precision of rating quality in the music assessment process. Expert judges (N = 23) rated a set of four recordings by middle school, high school, collegiate, and professional jazz big bands. A single common expert rater evaluated all 24 jazz ensemble performances. The data suggest that raters significantly vary in severity, items significantly vary in difficulty, and rating scale category structure significantly varies across raters. Implications for the improvement and management of rater quality in music performance assessment are provided.

Mitigating the Halo Effect: Managing the Wow Factor in Music Performance Assessments

This paper describes an ongoing project undertaken to assist assessors with recognising and ameliorating the influence of the halo effect when multiple criteria are applied in a rubric to score music performance assessments. The assessment context described in this chapter is set within the Victorian Certificate of Education (VCE), and specifically the VCE Music Investigation study. This VCE study provides a flexible pathway for students to perform, compose and research diverse traditional, contemporary, classical and popular music styles using instruments ranging from voice, tabla, recorder and ud to orchestral and wind-band instruments, instruments typically found in rock bands and digital instruments in solo and ensemble performances. This diversity poses issues of equity and rigour that must be addressed in practice through valid assessment tasks and the reliable application of assessment criteria. To meet this end, these tasks and criteria must be coupled with appropriate assessor training, guidance and documentation.

An Analysis of Ratings Assigned to String Musicians’ Performances at Solo and Ensemble Music Performance Assessment Events

The purpose of this study was to identify the influence of event type (solo and ensemble), school level (middle and high school), and event level (district and state) on overall performance ratings assigned to string musicians attending solo and ensemble festivals. Data consisted of participants’ (N = 10,275) ratings from district - and state-level solo and ensemble festivals sponsored by the Florida Orchestra Association from 2010 to 2014. Participants performed more solo events (n = 6,745) than ensembles (n = 3,530). Performances by middle school string players (n = 5,403) outnumbered those from high school (n = 4,872). Ninety-five percent of all participants (n = 9,761) earned a I/Superior or II/Excellent rating. Solo performances received more favorable ratings than ensemble performances at district festivals. However, ensembles were assigned more favorable ratings at state festivals. Evaluations of middle and high school participants’ performances were similar. District festival performances received more favorable ratings than state festival performances. With the increased importance of student data, student participation at adjudicated festivals, and providing a positive musical experience for students, directors may use these results to help their students receive beneficial assessments that influence long-term success.

Rater fairness in music performance assessment: Evaluating model-data fit and differential rater functioning

The purpose of this study was to investigate model-data fit and differential rater functioning in the context of large group music performance assessment using the Many-Facet Rasch Partial Credit Measurement Model. In particular, we sought to identify whether or not expert raters’ ( N = 24) severity was invariant across four school levels (middle school, high school, collegiate, professional). Interaction analyses suggested that differential rater functioning existed for both the group of raters and some individual raters based on their expected locations on the logit scale. This indicates that expert raters did not demonstrate invariant levels of severity when rating subgroups of ensembles across the four school levels. Of the 92 potential pairwise interactions examined, 14 (15.2%) interactions were found to be statistically significant, indicating that 10 individual raters demonstrated differential severity across at least one school level. Interpretations of meaningful systematic patterns emerged for some raters after investigating individual pairwise interactions. Implications for improving the fairness and equity in large group music performance evaluations are discussed.

An empirical study of a solo performance assessment model

The purpose of this study was to test a hypothesized model of solo music performance assessment. Specifically, this study investigates the influence of technique and musical expression on perceptions of overall performance quality. The Aural Musical Performance Quality (AMPQ) measure was created to measure overall performance quality, technique, musical expression, tone, intonation, rhythmic accuracy, articulation, tempo, dynamics, tone character, and interpretation. The factors of interest were identified from previous studies on musical performance constructs, musical achievement, musical expression, and scale construction. Alpha reliability for the 44-item Likert scale AMPQ measure was reported at .977. Volunteer adjudicators ( N = 58) were recruited to evaluate four solo instrumental recordings (brass, woodwind, voice, and string). A path analysis of the 232 performance evaluations provided empirical support for the hypothesized model. Technique demonstrated significant direct effects on assessments of overall performance quality and musical expression. Musical expression also demonstrated a significant direct effect on overall perceptions of performance quality. The results of this study are consistent with hypothesized model of performer-controlled musical factors.

Music Performance Assessment

Assessing student performance is a central challenge for music educators. In alignment with previous research, this article asserts that rubrics provide a viable and useful structure for assessing music performance. To expand the potential of rubrics in music education, challenges to effective rubric construction are identified and addressed through the construction of three different types of performance rubrics: (1) a discrete-component rubric, (2) an integrated-component rubric, and (3) a self-reflection rubric. Through analysis of these rubrics, we identify ways to construct assessment criteria that are sufficiently descriptive to accurately measure achievement and provide helpful feedback to learners. We demonstrate how criteria can be constructed that both encourage achievement of curriculum expectations and technical proficiency and leave room for students’ expressive intentions. We offer ideas for integrating these rubrics into the music classroom.

"A Woman of Genius is but a Heavenly Lunatic": Stoddard's Subversive Use of Music in The Morgesons

"A Woman of Genius is but a Heavenly Lunatic":Stoddard's Subversive Use of Music in The Morgesons Doris Davis (bio) I respect the mosquito; he is apparently an insignificance, but he sings and stings satanically. I am but a small bit myself; so I admire power in smallness. —Elizabeth Stoddard, August 20, 18551 In an early chapter of Elizabeth Stoddard's 1862 novel, The Morgesons, three vivacious old ladies assert in unison: "Miss Cassy don't play [the piano]. . . . Miss Verry's sun puts out her fire" (62).2 Their enthusiastic assessment of Verry's musical performance, rendered in metaphor, underscores a duality in the novel and points to musical skill as one basis of comparison. Stoddard offers here two distinct delineations of female musicianship: that of Cassy (or Cassandra), the novel's protagonist and narrator, in the form of the publicly sanctioned performer and singer in the parlor; and that of Verry (or Veronica), her younger sister, in the form of privately realized musical genius. Although music plays an integral role in the work, with almost forty references in the text, little critical analysis has been offered concerning its importance. Scholars have not considered this difficult novel in light of nineteenth-century attitudes toward female musical performance, composition, or genius. This paper discusses these conceptualizations and considers how Stoddard uses them in depicting her female characters' search for agency and self-expression. Additionally, it examines how the text's musical references contribute to its strikingly proto-modernist style. Often subversive in her use of music, Stoddard, like the mosquito referenced in the opening epigraph, both sings and stings throughout her self-styled recitative. Best known in her lifetime for nearly fifty short stories and poetry printed in various periodicals as well as a newspaper column in the Daily Alta California, Stoddard also published three novels in the 1860s. All but disappearing from the American literary scene for nearly a century, she was brought into public view once more by Lawrence Buell and Sandra A. Zagarell's 1984 critical edition of The Morgesons, followed by the inclusion of her story "Lemorne versus Huell" in several anthologies, including the Norton Anthology of American Literature, in which she is described by the editors as being "[s]omewhat ahead of her time" (Levine and Krupat 2524). Indeed, of mid-nineteenth-century American writers, few produced works as enigmatic and baffling as those conceived by Stoddard. Marked by [End Page 19] irony, cryptic dialogue, elliptical phrasing, and abrupt shifts in narrative, her fiction contains stylistic features most evocative of Emily Dickinson's poetry and of the later movement of modernism. In a now well-known comment made after he visited Dickinson in Amherst, Thomas Wentworth Higginson wrote his wife that she could fathom Dickinson's home life if she had read Stoddard's work (Buell and Zagarell xii). As Higginson's comment suggests, Dickinson and Stoddard share a number of parallels. Both were products of well-educated, prominent New England families living in small but established towns and both attended female seminaries, rejected evangelical Protestantism, and evinced a certain rebelliousness in work marked by an ironic façade. Although Dickinson demanded a secluded world and Stoddard a public arena, both incorporate music—particularly piano music—into their work in intriguing ways.3 Such inclusion in an era that considered piano study a valuable part of a young woman's upbringing is understandable; however, each suggests a departure from nineteenth-century acceptable patterns. Much of Dickinson's musical imagery suggests the power of music to shock. Music "stun[s]" us "With Bolts of Melody!" ("Poem 505," 23-24). Music, in the intensity of its beauty as either Robin song or piano, may be too much to bear. "I thought," Dickinson's persona reasons, "if I could only live / Till that first Shout got by - / Not all Pianos in the Woods / Had power to mangle me—" ("Poem 348," 5-8). Despite its beauty, or perhaps because of it, the piano is a potential instrument of torture. Stoddard presents the piano in an equally ambivalent manner. In "Lemorne versus Huell," protagonist Margaret Huell seems compromised by her musical expertise as a piano teacher, both economically...

Assessment in Competition as a Tool for Developing High School Marching Bands

Abstract This article outlines an investigation of assessment in competition as a tool to assist the development of marching bands. Based on the process model of assessing musical performance ( McPherson & Thompson, 1998 ), this study examined factors that impact judges’ assessment particularly in a competition setting and how the judging system can be used to improve the quality of Malaysian school bands. A qualitative analysis of transcripts of interviews with six marching band judges identified key issues relating to the ways experienced examiners use criteria in music performance assessment.

Understanding and Developing Rubrics for Music Performance Assessment

A primary difficulty with music performance assessment is managing its subjective nature. To help improve objectivity, rubrics can be used to develop a set of guidelines for clearly assessing student performance. Moreover, rubrics serve as documentation for student achievement that provides music teachers with a written form of accountability. This article examines the complexities of music performance assessment and provides an argument for the benefit of rubrics in the assessment process. In addition, discussion includes an overview of the various types of rubrics as well as suggestions for choosing and writing rubrics to assess musical performances.

Spectral and Temporal Measures in Hybrid Cochlear Implant Users

Compare auditory performance of Hybrid and standard cochlear implant users with psychoacoustic measures of spectral and temporal sensitivity and correlate with measures of clinical benefit. Cross-sectional study. Tertiary academic medical center. Hybrid cochlear implant users between 12 and 33 months after implantation. Hybrid recipients had preservation of low-frequency hearing. Administration of psychoacoustic, music perception, and speech reception in noise tests. Performance on spectral-ripple discrimination, temporal modulation detection, Schroeder-phase discrimination, Clinical Assessment of Music Perception, and speech reception in steady-state noise tests. Clinical Assessment of Music Perception pitch performance at 262 Hz was significantly better in Hybrid users compared with standard implant controls. There was a near significant difference on speech reception in steady-state noise. Surprisingly, neither Schroeder-phase discrimination at 2 frequencies nor temporal modulation detection thresholds across a range of frequencies revealed any advantage in Hybrid users. This contrasts with spectral-ripple measures that were significantly better in the Hybrid group. The spectral-ripple advantage was preserved even when using only residual hearing. These preliminary data confirm existing data demonstrating that residual low-frequency acoustic hearing is advantageous for pitch perception. Results also suggest that clinical benefits enjoyed by Hybrid recipients are due to improved spectral discrimination provided by the residual hearing. No evidence indicated that residual hearing provided temporal information beyond that provided by electric stimulation.

Ecological development and validation of a music performance rating scale for five instrument families

This study investigated ways to improve the quality of music performance evaluation in an effort to address the accountability imperative in tertiary music education. An enhanced scientific methodology was employed incorporating ecological validity and using recognized qualitative methods involving grounded theory and quantitative methods involving confirmatory factor analyses. By distilling the disciplinary consensus, this approach enabled the specific definition of the constructs and standards used in university student classical music performance examinations, and provided their refinement with the empirical development of standards-based, criterion-specific rating scales for five instrument families. The study found that the examiners in each instrument family used between 15 to 17 constructs and associated descriptors, and at least two general dimensions referring to technical proficiency and another relating to musicality and interpretation. Analyses showed acceptable internal consistency and construct validity for the scales. Findings suggested that, although several construct and general dimension commonalities were found among the items across all scales, the presence of significant instrument-specific differences suggested that the use of generic rating scales may not provide sufficient content validity. Implications for the application of standards in music performance assessment were discussed.

Reliability and Perceived Pedagogical Utility of a Weighted Music Performance Assessment Rubric

The purpose of this study was to investigate the reliability and perceived pedagogical utility of a multidimensional weighted performance assessment rubric used in Kansas state high school large-group festivals. Data were adjudicator rubrics ( N = 2,016) and adjudicator and director questionnaires ( N = 515). Rubric internal consistency was moderately high (.88). Dimension reliability ranged from moderately low ( W = .47) to moderate ( W = .77). Total score reliability was moderately high ( W = .80) and rating reliability was moderate ( W = .72). Findings suggested that reliability on the whole was within the range of previously researched music performance assessment tools. Questionnaire results suggested that the rubric provided a better instrument for justifying ratings and more detailed descriptions of what constituted acceptable performances than previously researched nonrubric forms; hence, adjudicators and directors perceived the rubric as possessing improved pedagogical utility.

Performer, Rater, Occasion, and Sequence as Sources of Variability in Music Performance Assessment

This study examined performer, rater, occasion, and sequence as sources of variability in music performance assessment. Generalizability theory served as the study's basis. Performers were 8 high school wind instrumentalists who had recently performed a solo. The author audio-recorded performers playing excerpts from their solo three times, establishing an occasion variable. To establish a rater variable, 10 certified adjudicators were asked to rate the performances from 0 (poor) to 100 (excellent). Raters were randomly assigned to one of five performance sequences, thus nesting raters within a sequence variable. Two G (generalizability) studies established that occasion and sequence produced virtually no measurement error. Raters were a strong source of error. D (decision) studies established the one-rater, one-occasion scenario as unreliable. In scenarios using the generalizability coefficient as a criterion, 5 hypothetical raters were necessary to reach the .80 benchmark. Using the dependability index, 17 hypothetical raters were necessary to reach .80.

The Influence of Teaching-Career Level and Primary Performance Instrument on the Assessment of Music Performance

The purpose of this investigation was to examine the influence of teaching-career level and primary instrument on music teachers' assessment of music performance. The main and interaction effects of three career-level conditions (in-service teachers, upper-division undergraduate students, and lower-division undergraduate students) were examined, along with two primary instrument conditions (brass, not brass), on tone, intonation, melodic accuracy, rhythmic accuracy, tempo, interpretation, and technique/articulation. Participants ( N=150) listened to performances of six junior high trumpet players of various abilities and rated them using the Woodwind Brass Solo Evaluation Form (Saunders & Holahan, 1997). No statistically significant differences were found for the vast majority of interactions or main effects for either career level or instrument condition, suggesting that no relationship exists between teaching-career level and primary performance instrument on the evaluation of music performances. June 1, 2004 October 27, 2004.

Editorial

This issue – the first of our 21st anniversary year – has two clear themes: the teaching and learning of world musics, and the assessment of musical performance and understanding. Within these themes, the papers present accounts of research at all levels of music teaching, from nursery to higher education, and range across diverse geographical contexts: Australia, China, the UK, the USA and Zambia. There is evidence here of a wide-ranging research community in music education, which would have been hard to imagine when the British Journal of Music Education was founded in 1984.