Human Evaluation Research Articles

Automatically Predicting Perceived Conversation Quality in a Pediatric Sample Enriched for Autism.

Social interaction quality ratings derived from short natural conversations can differentiate children with and without autism at the group level. In this work, we explored conversations between children and an unfamiliar adult who rated their social interaction success on six dimensions. Using hand-crafted acoustic and lexical features, we built different classifiers to predict children's dimensional conversation quality. The best classifier achieved 61% accuracy, which outperformed human raters (49%). Follow-up analyses revealed that a subset of features determined communication quality scores. Additionally, we extracted acoustic features using a pretrained audio transformer and improved our prediction to 68%. This study suggests that automatically predicting conversation quality could be an inexpensive and objective way to monitor intervention progress in children with communication challenges, and could be used to identify intervention targets for improving conversational success.

Measuring hand movement for suturing skill assessment: A simulation-based study

BackgroundTo maximize patient safety, surgical skills education is increasingly adopting simulation-based curricula for formative skills assessment and training. However, many standardized assessment tools rely on human raters for performance assessment, which is resource-intensive and subjective. Simulators that provide automated and objective metrics from sensor data can address this limitation. We present an instrumented bench suturing simulator, patterned after the clock face radial suturing model from the Fundamentals of Vascular Surgery, for automated and objective assessment of open suturing skills. MethodsFor this study, 97 participants (35 attending surgeons, 32 residents, and 30 novices) were recruited at national vascular conferences. Automated hand motion metrics, especially focusing on rotational motion analysis, were developed from the inertial measurement unit attached to participants’ hands, and the proposed suite of metrics was used to differentiate between the skill levels of the 3 groups. ResultsAttendings’ and residents’ performances were found to be significantly different from novices for all metrics. Moreover, most of our novel metrics could successfully distinguish between finer skill differences between attending and resident groups. In contrast, traditional operative skill metrics, such as time and path length, were unable to distinguish attendings from residents. ConclusionThis study provides evidence for the effectiveness of rotational motion analysis in assessing suturing skills. The suite of inertial measurement unit-based hand motion metrics introduced in this study allows for the incorporation of hand movement data for suturing skill assessment.

Deep learning-based automated lesion segmentation on mouse stroke magnetic resonance images

Magnetic resonance imaging (MRI) is widely used for ischemic stroke lesion detection in mice. A challenge is that lesion segmentation often relies on manual tracing by trained experts, which is labor-intensive, time-consuming, and prone to inter- and intra-rater variability. Here, we present a fully automated ischemic stroke lesion segmentation method for mouse T2-weighted MRI data. As an end-to-end deep learning approach, the automated lesion segmentation requires very little preprocessing and works directly on the raw MRI scans. We randomly split a large dataset of 382 MRI scans into a subset (n = 293) to train the automated lesion segmentation and a subset (n = 89) to evaluate its performance. We compared Dice coefficients and accuracy of lesion volume against manual segmentation, as well as its performance on an independent dataset from an open repository with different imaging characteristics. The automated lesion segmentation produced segmentation masks with a smooth, compact, and realistic appearance that are in high agreement with manual segmentation. We report dice scores higher than the agreement between two human raters reported in previous studies, highlighting the ability to remove individual human bias and standardize the process across research studies and centers.

Comparing Machine Learning Models and Human Raters When Ranking Medical Student Performance Evaluations.

The Medical Student Performance Evaluation (MSPE), a narrative summary of each student's academic and professional performance in US medical school is long, making it challenging for residency programs evaluating large numbers of applicants. To create a rubric to assess MSPE narratives and to compare the ability of 3 commercially available machine learning models (MLMs) to rank MSPEs in order of positivity. Thirty out of a possible 120 MSPEs from the University of Central Florida class of 2020 were de-identified and subjected to manual scoring and ranking by a pair of faculty members using a new rubric based on the Accreditation Council for Graduate Medical Education competencies, and to global sentiment analysis by the MLMs. Correlation analysis was used to assess reliability and agreement between student rank orders produced by faculty and MLMs. The intraclass correlation coefficient used to assess faculty interrater reliability was 0.864 (P<.001; 95% CI 0.715-0.935) for total rubric scores and ranged from 0.402 to 0.768 for isolated subscales; faculty rank orders were also highly correlated (rs=0.758; P<.001; 95% CI 0.539-0.881). The authors report good feasibility as the rubric was easy to use and added minimal time to reading MSPEs. The MLMs correctly reported a positive sentiment for all 30 MSPE narratives, but their rank orders produced no significant correlations between different MLMs, or when compared with faculty rankings. The rubric for manual grading provided reliable overall scoring and ranking of MSPEs. The MLMs accurately detected positive sentiment in the MSPEs but were unable to provide reliable rank ordering.

A Multi-Strategy Computer-Assisted EFL Writing Learning System With Deep Learning Incorporated and Its Effects on Learning: A Writing Feedback Perspective

Language learning has increasingly benefited from Computer-Assisted Language Learning (CALL) technologies, especially with Artificial Intelligence involved in recent years. CALL in writing learning acknowledged as the core of language learning is being realized by technologies like Automated Writing Evaluation (AWE), and Automated Essay Scoring (AES), which have developed considerably in both computer and language education fields. AWE has effectively enhanced EFL students’ writing performance to some extent, but such technology can only provide an evaluation in the form of scores, the majority of which are based on holistic scoring, resulting in the inability to provide comprehensive and detailed content-based feedback. In order to provide not only the writing multiple trait-specific evaluation scores, but also detailed writing feedback, we proposed a computer-assisted EFL writing learning system incorporating the neural network models and a couple of semantic-based NLP techniques, MsCAEWL, which fully meets the requirements of writing feedback theory, i.e., multiple, continuous, timely, clear, and multi-aspect guidance interactive feedback. The results of comparison experiments with the AWE baseline models and human raters demonstrated the superiority and the high correlation contained by the proposed system. The independent-sample t-test and paired-sample t-test results of the experiments on MsCAEWL effect validation suggested the significant impact of our proposed system in enhancing students’ EFL writing proficiency.

ChatGPT's Ability to Assess Quality and Readability of Online Medical Information: Evidence From a Cross-Sectional Study.

Introduction Artificial Intelligence (AI) platforms have gained widespread attention for their distinct ability to generate automated responses to various prompts. However, its role in assessing the quality and readability of a provided text remains unclear. Thus, the purpose of this study is to evaluate the proficiency of the conversational generative pre-trained transformer (ChatGPT) in utilizing the DISCERN tool to evaluate the quality of online content regarding shock wave therapy for erectile dysfunction. Methods Websites were generated using a Google search of "shock wave therapy for erectile dysfunction" with location filters disabled. Readability was analyzed using Readable software (Readable.com, Horsham, United Kingdom). Quality was assessed independently by three reviewers using the DISCERN tool. The same plain text files collected were inputted into ChatGPT to determine whether they produced comparable metrics for readability and quality. Results The study results revealed a notable disparity between ChatGPT's readability assessment and that obtained from a reliable tool, Readable.com (p<0.05). This indicates a lack of alignment between ChatGPT's algorithm and that of established tools, such as Readable.com. Similarly, the DISCERN score generated by ChatGPT differed significantly from the scores generated manually by human evaluators (p<0.05), suggesting that ChatGPT may not be capable of accurately identifying poor-quality information sources regarding shock wave therapy as a treatment for erectile dysfunction. Conclusion ChatGPT's evaluation of the quality and readability of online text regarding shockwave therapy for erectile dysfunction differs from that of human raters and trusted tools. Therefore, ChatGPT's current capabilities were not sufficient for reliably assessing the quality and readability of textual content. Further research is needed to elucidate the role of AI in the objective evaluation of online medical content in other fields. Continued development in AI and incorporation of tools such as DISCERN into AI software may enhance the way patients navigate the web in search of high-quality medical content in the future.

Grading of lung adenocarcinomas with simultaneous segmentation by artificial intelligence (GLASS-AI)

Preclinical genetically engineered mouse models (GEMMs) of lung adenocarcinoma are invaluable for investigating molecular drivers of tumor formation, progression, and therapeutic resistance. However, histological analysis of these GEMMs requires significant time and training to ensure accuracy and consistency. To achieve a more objective and standardized analysis, we used machine learning to create GLASS-AI, a histological image analysis tool that the broader cancer research community can utilize to grade, segment, and analyze tumors in preclinical models of lung adenocarcinoma. GLASS-AI demonstrates strong agreement with expert human raters while uncovering a significant degree of unreported intratumor heterogeneity. Integrating immunohistochemical staining with high-resolution grade analysis by GLASS-AI identified dysregulation of Mapk/Erk signaling in high-grade lung adenocarcinomas and locally advanced tumor regions. Our work demonstrates the benefit of employing GLASS-AI in preclinical lung adenocarcinoma models and the power of integrating machine learning and molecular biology techniques for studying the molecular pathways that underlie cancer progression.

Magnetic resonance imaging datasets with anatomical fiducials for quality control and registration

Tools available for reproducible, quantitative assessment of brain correspondence have been limited. We previously validated the anatomical fiducial (AFID) placement protocol for point-based assessment of image registration with millimetric (mm) accuracy. In this data descriptor, we release curated AFID placements for some of the most commonly used structural magnetic resonance imaging datasets and templates. The release of our accurate placements allows for rapid quality control of image registration, teaching neuroanatomy, and clinical applications such as disease diagnosis and surgical targeting. We release placements on individual subjects from four datasets (N = 132 subjects for a total of 15,232 fiducials) and 14 brain templates (4,288 fiducials), totalling more than 300 human rater hours of annotation. We also validate human rater accuracy of released placements to be within 1 – 2 mm (using more than 45,000 Euclidean distances), consistent with prior studies. Our data is compliant with the Brain Imaging Data Structure allowing for facile incorporation into neuroimaging analysis pipelines.

Image Perceptual Similarity Metrics for the Assessment of Basal Cell Carcinoma.

Efficient management of basal cell carcinomas (BCC) requires reliable assessments of both tumors and post-treatment scars. We aimed to estimate image similarity metrics that account for BCC's perceptual color and texture deviation from perilesional skin. In total, 176 clinical photographs of BCC were assessed by six physicians using a visual deviation scale. Internal consistency and inter-rater agreement were estimated using Cronbach's α, weighted Gwet's AC2, and quadratic Cohen's kappa. The mean visual scores were used to validate a range of similarity metrics employing different color spaces, distances, and image embeddings from a pre-trained VGG16 neural network. The calculated similarities were transformed into discrete values using ordinal logistic regression models. The Bray-Curtis distance in the YIQ color model and rectified embeddings from the 'fc6' layer minimized the mean squared error and demonstrated strong performance in representing perceptual similarities. Box plot analysis and the Wilcoxon rank-sum test were used to visualize and compare the levels of agreement, conducted on a random validation round between the two groups: 'Human-System' and 'Human-Human.' The proposed metrics were comparable in terms of internal consistency and agreement with human raters. The findings suggest that the proposed metrics offer a robust and cost-effective approach to monitoring BCC treatment outcomes in clinical settings.

Real Face Analysis of Masculinity-Femininity, Facial Ratios, and Attractiveness.

Previous studies have evaluated the influence of facial features in determining male and female sex using prototypical renderings or artificially altered faces in relatively small sample sizes. Using a large set of human photographs and raters, this study hypothesized that certain anatomic facial ratios are associated with perceptions of masculinity/femininity, can interact to predict sex, and are associated with ratings of attractiveness differently in males and females. Ratings of masculinity-femininity and binary self-identification (male or female) were compared with facial anatomic ratios from 827 frontal facial photographs. Ratios were used to characterize facial feature relativity, where higher ratio scores indicated relatively more facial feature representation. Femininity was associated with prominent middle third ratio, nose length, lip vermillion height, eye height, and eye width ratios; masculinity was associated with prominence of the upper and lower facial thirds, nose width, chin height, and philtrum height ratios (all P < 0.01). Subgroup analysis demonstrated many of these relationships persisted when evaluating masculinity in females and femininity in males. Misgendering in males was associated with greater middle third ratio and upper lip ratio, whereas misgendering in females was associated with increased nose width ratio. This study demonstrates associations of femininity with increased horizontal middle third representation, and masculinity was associated with increased upper and lower horizontal representation. These facial ratios interact to predict male and female sex, which could have implications for optimizing facial feminization/masculinization outcomes and building algorithms for artificial intelligence analysis of faces.

Human Versus Machine

This article implements a method for classifying adverse incidents involving reusable medical devices according to their underlying cause and assesses the level of agreement between different raters. To achieve this, the adverse incidents were classified into 1 or more of 62 separate categories, and the level of agreement between 3 experienced human raters was established. Moreover, the ChatGPT artificial intelligence tool was used to replicate the classification process used by human raters. The results showed that there was a fair level of agreement between human raters and a slight agreement between human raters and ChatGPT. This suggests that, although ChatGPT can intelligently classify adverse incidents, it was not able to replicate the performance of experienced human raters when given access to only the limited incident details and classification categories as provided for in this study.

Exploring the Efficacy of Automated Essay Scoring Systems in the Context of Korean EFL High School

Despite growing attention on the Automated Essay Scoring (AES) systems, it is difficult to find their actual use in Korean EFL high schools. Among various free and easily accessible AES systems available, this study aims to investigate the efficacy of three AES systems in the Korean EFL high school context: Grammarly, MS Editor, and ProWritingAid. To this end, the three AES systems were compared with two human raters in three aspects: scores reported on Korean EFL high school students’ writings, errors detected per error category, and errors detected per student writing. The t-test, ANOVA, and correlation analysis results have led to questions and doubts on the validity of the AES programs and their role as essay evaluators. While AES programs have shown growth, the AES programs are not yet comparable to human raters and teacher feedback. However, the AES programs do show potential as a supplementary tool in the Korean EFL context. Thus, a comparison of the three AES systems was examined in the hopes of finding the optimal choice of AES system. The findings provide information and insights on how AES systems may be implemented in Korean EFL high school classrooms, and further emphasized the importance of teachers’ role in implementing a new tool.

Evaluating robotic-assisted partial nephrectomy surgeons with fully convolutional segmentation and multi-task attention networks.

We use machine learning to evaluate surgical skill from videos during the tumor resection and renography steps of a robotic assisted partial nephrectomy (RAPN). This expands previous work using synthetic tissue to include actual surgeries. We investigate cascaded neural networks for predicting surgical proficiency scores (OSATS and GEARS) from RAPN videos recorded from the DaVinci system. The semantic segmentation task generates a mask and tracks the various surgical instruments. The movements from the instruments found via semantic segmentation are processed by a scoring network that regresses (predicts) GEARS and OSATS scoring for each subcategory. Overall, the model performs well for many subcategories such as force sensitivity and knowledge of instruments of GEARS and OSATS scoring, but can suffer from false positives and negatives that would not be expected of human raters. This is mainly attributed to limited training data variability and sparsity.

The role and features of peer assessment feedback in college English writing.

Peer assessment is a common pedagogical practice for evaluating students' writing in college English classrooms. However, in-depth research on the learning outcomes after peer assessment is scarce and inconsistent; how peer feedback is used has not been fully explored either. This study compared peer feedback to teacher feedback and explored the different features of feedback and its impact on draft revision. Two specific research questions were answered in this study: (1) In which aspects can peer feedback supplement teacher feedback in improving the linguistic features in writing? (2) What are the differences in features of peer feedback versus teacher feedback? And how do they connect to feedback take-in? Two writing tasks were assigned to 94 students. One received teacher feedback and the other peer feedback. Pre-feedback and post-feedback writings in both tasks (4 sets in total) were scored and human ratings were adjusted using Many-Facet Rasch modeling to control for differences in leniency. Drawing on three natural language processing (NLP) tools, this study also assessed writing qualities by comparing 22 selected indices related to the scoring rubrics for human raters, which involve three dimensions: cohesion, lexical quality and syntactic complexity. Peer and teacher feedback was then coded based on features of feedback to explore their influence on draft revision. The results showed that both peer and teacher feedback had positive effects on rating scores. We confirmed peer feedback as an effective classroom approach to improve writing, though limited compared to teacher feedback as reflected in the indices. In terms of feedback features, students often stopped at identifying the language problems, while the teacher provided more explanations, solutions or suggestions regarding the problems identified. Implications for peer feedback research and implementation of peer assessment activities are provided.

Beyond semantic distance: Automated scoring of divergent thinking greatly improves with large language models

Automated scoring for divergent thinking (DT) seeks to overcome a key obstacle to creativity measurement: the effort, cost, and reliability of scoring open-ended tests. For a common test of DT, the Alternate Uses Task (AUT), the primary automated approach casts the problem as a semantic distance between a prompt and the resulting idea in a text model. This work presents an alternative approach that greatly surpasses the performance of the best existing semantic distance approaches. Our system, Ocsai, fine-tunes deep neural network-based large-language models (LLMs) on human-judged responses. Trained and evaluated against one of the largest collections of human-judged AUT responses, with 27 thousand responses collected from nine past studies, our fine-tuned large-language-models achieved up to r = 0.81 correlation with human raters, greatly surpassing current systems (r = 0.12–0.26). Further, learning transfers well to new test items and the approach is still robust with small numbers of training labels. We also compare prompt-based zero-shot and few-shot approaches, using GPT-3, ChatGPT, and GPT-4. This work also suggests a limit to the underlying assumptions of the semantic distance model, showing that a purely semantic approach that uses the stronger language representation of LLMs, while still improving on existing systems, does not achieve comparable improvements to our fine-tuned system. The increase in performance can support stronger applications and interventions in DT and opens the space of automated DT scoring to new areas for improving and understanding this branch of methods.

Validating an Automated System of Evaluation for English Drafting Skill: A Case of Large-scale, High-stakes Selection of Entry-level Managerial Positions

The effectiveness of using a constructed response measure for assessing drafting ability is proven and has been used extensively in managerial selection in state-owned organizations. With the advent of online recruitment trends and technology-enhanced assessments, automated scoring has been conceived as a replacement for human scoring with the purpose of emulating the human scoring system. In the context of large-scale writing assessments, automated scoring could provide superior results to human scoring in terms of validity and reliability. In the present study, an attempt has been made to validate an automated essay scoring (AES) algorithm. The study was conducted on a sample of 11,497 randomly selected from a population of 54,392 shortlisted candidates for a national-level examination for the selection of entry-level executives in managerial positions in state-owned banks and a state-owned insurance company. The evaluation of the descriptive (constructed response) component of the examination (English composition) was carried out parallelly by four expert human raters and the AES algorithm. The parameters for evaluation were devised and made available to raters and the algorithm. Data were analysed using mean SD and Pearson correlation coefficient. Results show that the mean scores of human expert raters ( M = 14.648) and automated algorithm method ( M = 15.804) were similar. Further analysis was also undertaken to check the convergent validity of the features used in the algorithm by examining the relation of algorithm scores with sub scores from the objective test of the same construct. Results indicate a significant correlation. It can thus be said that the algorithm scoring method developed can be considered a replacement in that it can complement human expert raters in the evaluation of descriptive papers with consistent scoring and fairness, without inherent biases of inter-rater and intra-rater variation, in addition to practical benefits of speed and cost.

Digital enhancement of pronunciation assessment: Automated speech recognition and human raters*

Digital enhancement of pronunciation assessment: Automated speech recognition and human raters*

Measurement of interspinous motion in dynamic cervical radiographs using a deep learning-based segmentation model.

Interspinous motion (ISM) is a representative method for evaluating the functional fusion status following anterior cervical discectomy and fusion (ACDF) surgery, but the associated measuring difficulty and potential errors in the clinical setting remain concerns. The aim of this study was to investigate the feasibility of a deep learning-based segmentation model for measuring ISM in patients who underwent ACDF surgery. This study is a retrospective analysis of flexion-extension dynamic cervical radiographs from a single institution and a validation of a convolutional neural network (CNN)-based artificial intelligence (AI) algorithm for measuring ISM. Data from 150 lateral cervical radiographs from the normal adult population were used to train the AI algorithm. A total of 106 pairs of dynamic flexion-extension radiographs from patients who underwent ACDF at a single institution were analyzed and validated for measuring ISM. To evaluate the agreement power between human experts and the AI algorithm, the authors assessed the interrater reliability using the intraclass correlation coefficient and root mean square error (RMSE) and performed a Bland-Altman plot analysis. They processed 106 pairs of radiographs from ACDF patients into the AI algorithm for autosegmenting the spinous process created using 150 normal population radiographs. The algorithm automatically segmented the spinous process and converted it to a binary large object (BLOB) image. The rightmost coordinate value of each spinous process from the BLOB image was extracted, and the pixel distance between the upper and lower spinous process coordinate value was calculated. The AI-measured ISM was calculated by multiplying the pixel distance by the pixel spacing value included in the DICOM tag of each radiograph. The AI algorithm showed a favorable prediction power for detecting spinous processes with an accuracy of 99.2% in the test set radiographs. The interrater reliability between the human and AI algorithm of ISM was 0.88 (95% CI 0.83-0.91), and its RMSE was 0.68. In the Bland-Altman plot analysis, the 95% limit of interrater differences ranged from 0.11 to 1.36 mm, and a few observations were outside the 95% limit. The mean difference between observers was 0.02 ± 0.68 mm. This novel CNN-based autosegmentation algorithm for measuring ISM in dynamic cervical radiographs showed strong agreement power to expert human raters and could help clinicians to evaluate segmental motion following ACDF surgery in clinical settings.

Development and validation of a quality control procedure for automatic segmentation of hippocampal subfields.

Automatic segmentation methods for in vivo magnetic resonance imaging are increasing in popularity because of their high efficiency and reproducibility. However, automatic methods can be perfectly reliable and consistently wrong, and the validity of automatic segmentation methods cannot be taken for granted. Quality control (QC) by trained and reliable human raters is necessary to ensure the validity of automatic measurements. Yet QC practices for applied neuroimaging research are underdeveloped. We report a detailed QC and correction procedure to accompany our validated atlas for hippocampal subfield segmentation. We document a two-step QC procedure for identifying segmentation errors, along with a taxonomy of errors and an error severity rating scale. This detailed procedure has high between-rater reliability for error identification and manual correction. The latter introduces at maximum 3% error variance in volume measurement. All procedures were cross-validated on an independent sample collected at a second site with different imaging parameters. The analysis of error frequency revealed no evidence of bias. An independent rater with a third sample replicated procedures with high within-rater reliability for error identification and correction. We provide recommendations for implementing the described method along with hypothesis testing strategies. In sum, we present a detailed QC procedure that is optimized for efficiency while prioritizing measurement validity and suits any automatic atlas.

Automated speech scoring of dialogue response by Japanese learners of English as a foreign language

ABSTRACT Purpose: This study builds a new system for automatically assessing learners’ speech elicited from an oral discourse completion task (DCT), and evaluates the prediction capability of the system with a view to better understanding factors deemed influential in predicting speaking proficiency scores and the pedagogical implications of the system. Methodology: We developed a system with a tripartite structure using an automatic speech recogniser, a set of modules that compute a number of speech features, and a scoring model. In total, 210 participants with intermediate English language proficiency level were administered a multi-turn oral DCT, a task which closely resembles discourse in real-life situations. The collected speech and transcribed files were eyeballed and rated by human raters first. Eighty percent of the original dataset was then used to train and examine our prediction model against the remainder of the dataset. Findings: The exact agreement between human and machine scores was 72%, moderately high, and comparable to the literature on automated speech scoring. It could provide a basis for deploying the system in a low-stakes practice environment. Originality/value: This study makes a unique contribution to the wider scholarship, where the use of single-turn DCTs remains prevalent, by presenting a new reliable scoring system for learner speech using an automated DCT with multiple turns. It offers useful insight into ways in which the system could be used in a low-stakes environment, including foreign language classroom settings.