Holistic Ratings Research Articles

Writing performance and discourse organization in L2 Chinese: A longitudinal case study

The present study explored the development of writing performance and discourse organization in Chinese as a second language (L2) writing, focusing on the effects of genre and learning context. Tracking an L2 Chinese learner’s written production in two genres (i.e., narrative and argumentative) across four learning contexts (i.e., pre-study abroad, study abroad, post-study abroad, and delayed post-study abroad) longitudinally over four years, this study revealed that holistic ratings and discourse organizational devices showed distinct patterns in different genres and across learning contexts. Narrative essays generally received lower ratings than argumentative essays. Ratings of narrative essays remained relatively stable across four learning contexts, whereas those of argumentative essays displayed an omega-shaped pattern over the four-year period. Among six discourse organizational devices, subordinate clauses, textual connectives, and grammatical metaphors were used more in argumentative essays, but coordinate clauses and topic-prominent constructions were used more in narrative essays. The six discourse organizational devices exhibited discrete developmental trends throughout the four years of learning. This study provides a deeper understanding of writing performance and discourse organization in L2 Chinese development.

Exploring the creativity-curiosity link in early childhood

Childhood is a pinnacle of both creativity and curiosity, and although these constructs theoretically overlap, few studies have probed whether they are directly related in childhood or driven by similar cognitive and emotional processes. Across two online Zoom sessions, 36 3- to 6 year-olds completed six tasks measuring diverse manifestations of curiosity and creativity, as well as tasks assessing vocabulary, self-esteem, and executive function. Caregivers also completed questionnaires regarding their children's curiosity. Only two significant, positive correlations were found between indices of creativity and curiosity: between originality of ideas (creativity) and breadth of exploration (curiosity), and between creativity on a production-based task and parent-reported breadth of exploration (curiosity). Further, the two constructs were predicted by different child characteristics. Age was the main predictor of creativity; originality of children's ideas in two divergent thinking tasks decreased with age, while fluency and holistic ratings of production-based tasks increased. Self-esteem, in turn, was the strongest predictor of curiosity, correlating positively with several subtypes of parent-reported curiosity. The results of this exploratory study suggest creativity and curiosity may not be as closely linked in childhood as some have proposed, and that pinpointing their relations will require careful attention to the individual components and expressions of each construct.

Exploring ChatGPT’s Clinical Ethics Ability: A pilot study

Background: Artificial Intelligence (AI), particularly public models like ChatGPT, has revolutionized the generation of human-like thought processes and text. Across healthcare, the integration of AI in decision-making processes is increasingly pervasive. However, the application of AI in ethical decision-making remains relatively unexplored. Methods: Ethics consultation notes from a tertiary academic medical center were de-identified. We trained ChatGPT using three separate ‘chats’ with one, two, or five unique notes and asked it to produce an ethical analysis/discussion and recommendations for a test case. We conducted this same series again but gave ChatGPT only the ethical analysis/discussion and recommendation sections from the training notes to learn from. Two independent raters scored ChatGPT’s ethics consultation documentation using the validated Ethics Consult Quality Assessment Tool (ECQAT). Results: When trained with full notes ChatGPT’s ECQAT overall holistic rating score for each ‘chat’ was 2.5 for one note, 1.5 for two, and 2.5 for five. When trained using only the ethical analysis/discussion and recommendation sections, ChatGPT scored 3 for one note, 2 for two, and 1 for five. Conclusion: ChatGPT's variable performance, influenced by training data, highlights its poor baseline ability and the need for targeted training. While initial improvement was observed with example consultations, complexity and scale affected performance adversely. The findings emphasize the importance of human oversight, as ChatGPT alone is unable to match human expertise. ChatGPT does exhibit potential for substantial improvement with better training and further research is needed to successfully make use of this powerful and widely accessible tool.

Predicting functional adequacy from complexity, accuracy, and fluency of second-language picture-prompted speaking

Functional adequacy (FA) is a construct of task achievement in communicative settings and focuses on the extent to which task requirements are satisfied by effectively conveying intended messages. Recent studies in second-language speaking and writing have emphasized the importance of FA in addition to complexity, accuracy, and fluency (CAF); FA and CAF are combined into the acronym CAFFA. This study aims to investigate (a) the extent to which CAF measures can explain FA holistic ratings in oral picture narration among Japanese learners of English; (b) how these results are moderated by different picture tasks; and (c) the comparability of means, variances, and correlations of the same FA ratings and CAF measures across tasks. Results of multiple regression analyses indicate that only a speed fluency measure (syllables per minute) significantly predicts FA, while a substantial proportion of FA remains unexplained by CAF, highlighting the separate and related nature of the two constructs. Moreover, the prediction of FA by CAF is consistent across picture tasks, with means and variances of FA and CAF measures being generally comparable, and with correlations of the same measures across tasks not being consistently strong, except for syllables per minute. The paper discusses implications and offers suggestions for future research.

COMPARATIVE ANALYSIS OF REGRESSION ALGORITHMS FOR PREDICTING STUDENTS' OVERALL RATINGS

In the rapidly evolving landscape of education, the integration of Artificial Intelligence (AI) has revolutionized the way we assess students. This research study focuses on the development and evaluation of a modern Machine Learning (ML) model designed to allocate comprehensive ratings to a diverse group of 10 students. The model's objective is to assess students not merely on their academic achievements but also on a holistic scale, taking into account a wide array of parameters, as detailed below. Various factors were systematically analyzed and integrated, including academic performance, extracurricular activities, study routines, and personal attributes, among others. This comprehensive approach seeks to provide a more nuanced and equitable representation of a student's overall capabilities and potential. To determine the most effective ML regression algorithm for predicting these holistic ratings, a rigorous comparative analysis was conducted, evaluating the performance of multiple algorithms. The study considered factors such as prediction accuracy, model interpretability, and computational efficiency. The findings of this research shed light on which ML regression algorithm best predicts the overall ratings for the students in question, offering insights into the potential for AI-driven holistic assessment in educational settings. By embracing AI technologies, educators and institutions can enhance their understanding of students, ultimately contributing to more tailored and equitable educational experiences. This research carries implications not only for educational practitioners but also for policymakers and technologists seeking to leverage AI for educational improvement. The detailed methodology, results, and practical implications of this study are discussed in the full research paper. It contributes to the ongoing dialogue on the integration of AI in education and its potential to create more inclusive and effective learning environments.

Measuring the Concurrent Validity of the Athletic Skills Track Using Expert Judgment

Purpose: The aim of this study is to measure the concurrent validity of the Athletic Skills Track (AST) by examining whether its outcome score correlates with the holistic judgments of experts about the quality of movement. Method: Video recordings of children performing the AST were shown to physical education teachers who independently gave a holistic rating of the movement quality of each child. Results: Both intra- and interrater reliability of the teachers’ ratings were moderate to good. The holistic judgments on movement quality were significantly correlated with AST time, showing that higher ratings were associated with less time required to complete the track. Next, hierarchical stepwise regression indicated that in addition to the holistic rating, also age, but not gender, explained part of the variance in AST time. Conclusion: The findings show that the AST has good concurrent validity and provides a fast, indirect indication for quality of movement.

Depth-Perception-Based Representation in Holistic Rating on ESL Essay Writing

This paper proposes to use depth perception to represent raters’ decision in holistic evaluation of ESL essays, as an alternative medium to conventional form of numerical scores. The researchers verified the new method’s accuracy and inter/intra-rater reliability by inviting 24 ESL teachers to perform different representations when rating 60 essays written by Chinese ESL learners. During numerical representation, the raters expressed their evaluation in the form of numbers, while with the DP-based method, the raters expressed their evaluation by marking distances with nail tags on a wood ruler with hidden scale marks. Then the researchers translated the distance results into numbers and compared the accuracy and inter/intra-rater consistency of the two approaches by referring to these essays’ criteria scores from eight expert raters. The results showed that DP-based method could improve raters’ performance, producing more similar results to experts’ scores, with higher consistency both among different raters and within the same individual rater.

Strategy use in a spoken dialog system–delivered paired discussion task: A stimulated recall study

With recent technological advances, researchers have begun to explore the potential use of spoken dialog systems (SDSs) for L2 oral communication assessment. While several studies support the feasibility of building these systems for various types of oral tasks, research on the construct validity of SDS-delivered tasks is still limited. Thus, this study examines the cognitive processes engaged by an SDS-delivered paired discussion task, focusing on strategic competence, an essential component of L2 oral communication ability. Thirty adult test-takers completed a paired discussion task with an SDS acting as an interlocutor and provided stimulated recalls about their strategy use in the task. Three trained raters independently evaluated the test-takers’ oral task responses using a holistic rating scale devised for the task. Findings revealed the use of six categories of construct-relevant strategies during task performance. While no statistically significant differences were found in the use of these categories between high- and low-ability test-takers, marked differences were observed in the use of individual strategies within the categories between the test-takers at the two levels. These findings provide insight into how test-takers at different ability levels cognitively interact with SDS-delivered paired discussion tasks and offer implications for the design and validation of such tasks.

숙련 및 미숙련 평가자들의 총체적, 분석적 및 진단적 채점기준을 활용한 영작문 평가 방식의 비교 분석 연구

Objectives This study aimed to investigate experienced and novice raters’ assessment behaviors when they used holistic, analytic, and diagnostic rating scales in ESL writing assessments. It also closed examined how the two groups of raters perceived the different rating scales.
 Methods The experienced and novice rater groups consisted of five native English-speaking professors with more than three years of experience teaching and assessing ESL writing and four native English-speaking professors with less than three years of experience teaching and assessing ESL writing. Both groups of raters assessed 30 ESL compositions using the three different rating scales and completed an assessor questionnaire.
 Results The findings indicated that the novice raters were more lenient in assessing ESL writing across the three rating scales. The experienced raters also showed more variance in their scores than the novice raters when they used the holistic and analytic (i.e., task fulfillment) rating scales. In contrast, the novice raters showed more variance in their scores than the experienced raters when they used the analytic (i.e., lexical and grammar) and diagnostic rating scales. While the experienced raters preferred the holistic rating scale, the novice raters preferred the analytic rating scale. Both groups liked the diagnostic rating scale the least.
 Conclusions Based on these findings, this study proposes that raters should consider the assessment purpose and context and rater variability when choosing an appropriate rating scale.

An exploratory study of the construct measured by automated writing scores across task types and test occasions

With the growing use of Automated Essay Scoring (AES) to score second language (L2) writing performance, questions are often raised concerning the interpretation of automated scores. Such questions are typically investigated by examining the association between automated scores and human holistic ratings at one point in time. However, this line of research cannot answer questions about whether and how this association varies across tasks and test occasions. This exploratory study addresses this gap by examining the association between automated writing scores and human multiple-trait ratings of essays written by 48 learners of English in response to TOEFL iBT independent and integrated writing tasks on two test occasions, before and after a period of English language study. Each essay (N= 192) was scored by e-rater and rated by a group of human raters on various writing features. The findings indicated that the associations between e-rater scores and human ratings of some writing features varied significantly across task types and tended to be stronger for essays written after, than for essays written before, English language study. We discuss the findings and their implications for future research on automated scoring in L2 writing assessment.

Building an Initial Validity Argument for Binary and Analytic Rating Scales for an EFL Classroom Writing Assessment: Evidence from Many-Facets Rasch Measurement

Although much research has compared the functioning between analytic and holistic rating scales, little research has compared the functioning of binary rating scales with other types of rating scales. This quantitative study set out to preliminarily and comparatively validate binary and analytic rating scales intended for use in formative assessment and for paragraph writing assessment in a Thai EFL university classroom context. Specifically, this study applied an argument-based validation approach to build an initial validity argument for the rating scales with emphasis on the evaluation, generalization, and explanation inferences, and employed a many-facets Rasch measurement (MFRM) approach to investigate the psychometric functionalities of the rating scales which served as the initial validity evidence for the rating scales. Three trained teacher raters applied the rating scales to rate the same set of 51 opinion paragraphs written by English-major students. The rating scores were analysed following the MFRM psychometrics. Overall, the MFRM results revealed that (1) the rating scales largely generated accurate writing scores, supporting the valuation inference, (2) the raters were self-consistent in applying the rating scales, contributing to the generalization inference, (3) the rating scales sufficiently captured the defined writing construct, substantiating the explanation inference, and (4) the binary rating scale showed more desirable psychometric properties than the analytic rating scale. The present findings confirm the appropriate functioning and reasonable validity argument of the rating scales and highlight the greater potential of the binary rating scale to mitigate rater inconsistency and cognitive load in a formative classroom assessment.

Structure and coherence as challenges in composition: A study of assessing less proficient EFL writers’ text quality

Students are usually expected to write full texts in English as a foreign language (EFL) at the end of secondary education. However, research on EFL writing at school is scarce, especially regarding less proficient writers, and seldom focuses on deep-level text features such as structure and coherence. Based on a sample of 166 EFL students in Year 9 attending German middle and lower performance track schools, this study examined 326 narrative and argumentative texts. First, we assessed structure and coherence via analytic ratings using detailed rubrics to gain insights into possible challenges for students. Our analysis showed that relevant text parts (such as the conclusion) were mostly missing and that students struggled to establish a broad common thread with argumentative texts being overall less structured and coherent than narrative texts. Second, we used the software Comproved® to conduct holistic ratings of overall text quality and compared them with our analytic ratings. Large correlations between both ratings suggest that structure and coherence are important aspects of text quality. We discuss how our rubrics can serve as a useful tool for assessment for learning and assist less proficient writers in establishing deep-level features in their texts. • We developed analytic rating rubrics to assess text structure and coherence. • Secondary EFL students especially struggle with text structure besides coherence. • Students often omit conclusions and do not adhere to a main idea. • Structure and coherence relate to text quality according to holistic ratings. • Analytic rating rubrics may be used as a tool for assessment for learning in class.

“We improved presenting voice in our writing”

Abstract This study investigated the role of teacher feedback in presenting students’ voice and writing quality of personal statements (PSs). PSs are essays intended to grant writers’ admission into graduate programs. Twenty third-year undergraduate students majoring in English participated in this study. Three drafts were collected in an English as a foreign language (EFL) writing class, and teacher feedback was given on the first two drafts. Three participants were interviewed to share their thoughts about voice development and revisions. Drafts 1 and 3 were analyzed using an analytic voice rubric, a holistic rating, and an analytic rubric. It was found that students presented a unique voice in Draft 3 and improved their writing quality by using teacher feedback in classroom contexts. Overall voice was found to strongly correlate with writing quality. Findings of the study demonstrate the importance of teacher feedback in improving students’ abilities to articulate their voice and writing quality in EFL writing.

Optimizing Clinical Reasoning Assessments With Analytic and Holistic Ratings: Examining the Validity, Reliability, and Cost of a Simplified Patient Note Scoring Procedure.

Post-standardized patient (SP) encounter patient notes used to assess students' clinical reasoning represent a significant time burden for faculty who traditionally score them. To reduce this burden, the authors previously reported a complex faculty-developed scoring method to assess patient notes rated by nonclinicians. The current study explored whether a simplified scoring procedure for nonclinician raters could further optimize patient note assessments by reducing time, cost, and creating additional opportunities for formative feedback. Ten nonclinician raters scored patient notes of 141 students across 5 SP cases by identifying case-specific patient note checklist items. The authors identified the bottom quintile of students using the proportion of correct items identified in the note (percent-scores) and case-specific faculty-generated scoring formulas (formula-scores). Five faculty raters scored a subset of notes from low, borderline, and high-performing students (n = 30 students) using a global rating scale. The authors performed analyses to gather validity evidence for percent-scores (i.e., relationship to other variables), investigate its reliability (i.e., generalizability study), and evaluate its costs (i.e., faculty time). Nonclinician percent- and formula-scores were highly correlated ( r = .88) and identified similar lists of low-performing students. Both methods demonstrated good agreement for pass-fail determinations with each other (Kappa = .68) and with faculty global ratings (Kappa percent =.61; Kappa formula =.66). The G-coefficient of percent-scores was .52, with 38% of variability attributed to checklist items nested in cases. Using percent-scores saved an estimated $746 per SP case (including 6 hours of faculty time) in development costs over formula-scores. Nonclinician percent-scores reliably identified low-performing students without the need for complex faculty-developed scoring formulas. Combining nonclinician analytic and faculty holistic ratings can reduce the time and cost of patient note scoring and afford faculty more time to coach at-risk students and provide targeted assessment input for high-stakes summative exams.

Raters’ perceptions of rating scales criteria and its effect on the process and outcome of their rating

It is widely believed that human rating performance is influenced by an array of different factors. Among these, rater-related variables such as experience, language background, perceptions, and attitudes have been mentioned. One of the important rater-related factors is the way the raters interact with the rating scales. In particular, how raters perceive the components of the scales to further plan their scoring seems important. For this aim, the present study investigated the raters’ perceptions of the rating scales and their subsequent rating behaviors for two analytic and holistic rating scales. Hence, nine highly experienced raters were asked to verbalize their thoughts while rating student essays using IELTS holistic scale and the analytic scale of ESL Composition Profile. Upon analyzing the think-aloud protocols, four themes emerged. The findings showed that when rating holistically, the raters either referred to the holistic scale components to validate their ratings (validation) or had a pre-evaluation reading to rate in a more reliable way (dominancy). In analytic rating, on the other hand, the raters used a pre-evaluation scale reading in order to keep the components and their criteria to memory to evaluate the text more accurately (dominancy) or regularly moved between the text and the scale components to assign a score (oscillation). Furthermore, the results of a Wilcoxon signed-rank test showed that when using the holistic and analytic rating scales, the raters assigned significantly different scores to the texts. On the whole, the results revealed that the way the raters perceived the scale components will affect their judgement of the texts. The study also provides several implications for rater training programs and EFL writing assessment.

An Investigation of the Impact of Jagged Profile on L2 Speaking Test Ratings: Evidence from Rating and Eye-tracking Data

ABSTRACT The factors that influence rater scoring have been a subject of great interest to researchers in second language assessment. However, the research on the impact of test-takers’ speech profiles (e.g., a jagged or a flat profile reflecting analytic subscores) on raters’ scoring behaviors remains to be seen. To investigate the role of speech profiles in scoring, we collected analytic and holistic rating scores from 28 trained raters while they were marking the performances of three groups of speakers with distinct profiles, determined by prior ratings. We tracked eleven of the raters’ eye-movements to record how often and how long they looked at the various categories on the rating scales. We found that the raters perceived speakers who have better pronunciation as overall more competent speakers. Meanwhile, speakers’ score profiles influenced raters’ attention: raters fixated longer and more often, and made more eye-visits, to the lexical grammar category while assessing speakers with a jagged profile. Raters spent less time assessing the pronunciation of the speakers who were pre-identified as having better pronunciation. The findings shed light on the impact of speech characteristics on raters’ cognition and score assignments and therefore have important implications for rater training in L2 speaking assessments.

Socio-cultural integration and holistic health among Indigenous young adults

BackgroundResearch on associations between social integration and wellbeing holds promise to inform policy and practice targets for health promotion. Yet, studies of social connection too frequently rely on overly simplistic measures and give inadequate attention to manifestation and meanings of social integration across diverse groups. We use the term socio-cultural integration to describe expanded assessment of both social and cultural aspects of belonging and connection.MethodsWe examined 7 distinct indicators of socio-cultural integration, identified heterogeneous patterns of responses across these indicators using latent profile analysis, and determined their relevance for wellbeing using survey data from a study with Indigenous communities in the U.S. and Canada. Wellbeing was measured using holistic ratings of self-rated physical, emotional, and spiritual health.ResultsLatent profile analysis (LPA) of responses to the 7 socio-cultural integration variables yielded a 3-class model, which we labeled low, moderate, and high integration. Mean scores on self-rated physical, mental and spiritual health were significantly associated with LPA profiles, such that those in the low integration group had the lowest self-rated health scores and those in the high integration group had the highest health scores. With the exception of similar ratings of cultural identification between low and moderate integration profiles, patterns of responses to the diverse socio-cultural integration measures varied significantly across the 3 latent profiles.ConclusionsResults underscore the importance of expanding our assessment of social integration with attention to the interrelationships of family, community, culture, and our environment. Such concepts align with Indigenous conceptions of wellbeing, and have relevance for health across cultures. More concretely, the indicators of socio-cultural integration used in this study (e.g., cultural identity, having a sense of connectedness to nature or family, giving or receiving social support) represent malleable targets for inclusion in health promotion initiatives.

Assessing the Quality of Argumentative Texts: Examining the General Agreement Between Different Rating Procedures and Exploring Inferences of (Dis)agreement Cases

Assessing argumentative writing skills is not a straightforward task, as multiple elements need to be considered. In function of providing feedback to students and keeping track of their progress, evaluating argumentative texts in a suitable, valid and efficient way is important. In this state-of-the-art exploratory study, 130 argumentative texts written by eleventh graders were assessed by means of three different rating procedures (i.e., absolute holistic rating, comparative holistic rating, and absolute analytic rating). The main aim of this study is twofold. First, we aim to examine the correlations between the three rating procedures and to study the extent to which these procedures differ in assigning scores. In doing so, the more innovative approach of pairwise comparisons is compared to more established assessment methods of absolute holistic and analytic rating. Second, we aim to identify key characteristics that determine the quality of an argumentative text, independent of the rating procedure used. Furthermore, key elements of mid-range, weak and strong argumentative texts were studied in detail. The results reveal low to moderate agreement between the different procedures, indicating that all procedures are suitable to assess the quality of an argumentative text; each procedure, however, has its own qualities and applicability.

Childminding professionalism and professionalisation in Ireland: A different story

This research focussed on documenting the praxis and paedagogy of paid, professional childminding (family childcare/day care) in Ireland. It explored professionalism and professionalisation among childminders in the context of the evolving understanding of professionalism in Early Childhood Education and Care (ECEC) nationally and internationally. The research was conducted within the framework of Ecocultural Theory (ECT) on the eve of mandatory regulation of childminding against the backdrop of Irish ECEC policy. A mixed method approach was adopted, using the Ecocultural Family Interview for Childminders (EFICh), including participants’ photographs, case study surveys, researcher field notes and holistic ratings. We present findings related to childminder professionalism and professionalisation, highlighting its significant differences from centre-based provision. It is vital to understand childminding as an ecocultural adaptation to create a stable family niche, relationally and economically. It helps to explain childminders’ approach to just-in-time training as adult learners, their desire for public recognition of childminding’s differences and unique value, and their need for supportive supervision in their family home ahead of annual inspection. Imposed professionalism is rejected in favour of a participatory approach sensitive to agentic childminders’ professional development. To support and develop professional 21st century childminding, Ireland requires a tailored regulatory system specific to childminding.

Predicting L2 Speaking Proficiency Using Syntactic Complexity Measures: A Corpus-Based Study

This paper discusses the syntactic complexity factors contributing to the achievement of a higher proficiency level in English speech. Here I have examined complexification at the sentential, clausal, phrasal and nominal level of syntactic organisation in a Korean learner spoken corpus using quantitative measures and compared the scores with holistic ratings of learners’ overall speaking quality. After the normality assumption analysis confirmed that the logistic regression was appropriate, an analysis was performed to ascertain the effects of complexity measures on participants’ L2 proficiency. First, length-based complexity features, namely, MLT and coordinated phrases, namely, CPT and CPC were found to be predictors for English speaking proficiency. Next, the logistic regression model was statistically significant and explained 36.3% of the variance in classification according to L2 proficiency and correctly classified 75.4% of cases. Results also showed that when learners come to use the coordinated phrases per clause proficiently, they were over 24 times more likely to achieve higher proficiency in spoken English. Finally, an effective equation was proposed to help educators classify EFL learners according to proficiency in L2 speech after gauging the selected complexity dimensions. However, more comprehensive studies which consider other methods of unit segmentation for spoken data or include more measures to predict L2 speech proficiency, are necessary to verify the results of this study. Keywords: Learner spoken corpus; Syntactic complexity; Monologue; EFL; Logistic regression