Human Graders Research Articles

Diagnostic Accuracy of Automated Diabetic Retinopathy Image Assessment Software: IDx-DR and RetCAD.

Automated diabetic retinopathy (DR) screening using artificial intelligence has the potential to improve access to eye care by enabling large-scale screening. However, little is known about differences in real-world performance between available algorithms. This study compares the diagnostic accuracy of two AI screening platforms, IDx-DR and RetCAD, for detecting referable diabetic retinopathy (RDR). Retinal images from 758 patients with diabetes were collected during screening from various clinics in Poland. Each patient was graded by three graders with 320 patients graded by Polish and 438 patients graded by Indian graders, with the majority decision serving as the reference standard. The images were evaluated independently by the IDx-DR and RetCAD algorithms. Sensitivity, specificity, positive and negative predictive values, and agreement between algorithms and human graders were calculated and statistically compared. IDx-DR demonstrated higher sensitivity of 99.3% but lower specificity of 68.9% for RDR detection compared to RetCAD which had 89.4% sensitivity and 94.8% specificity. The positive predictive value was higher for RetCAD (96.4% vs 48.1% for IDx-DR) while the negative predictive value was higher for IDx-DR (99.5% vs 83.1% for RetCAD). Both algorithms achieved high sensitivity (> 95%) for sight-threatening diabetic retinopathy detection. In this direct comparison using the same patient cohort, the two algorithms showed differences in their operating parameters for RDR screening. IDx-DR prioritized avoiding false negatives over false positives while RetCAD maintained a more balanced trade-off. These results highlight the variable performance of current artificial intelligence screening solutions and suggest the importance of considering algorithm performance metrics when deploying automated diabetic retinopathy screening programs, based on available healthcare resources.

FaceFinder: A Machine Learning Tool for Identification of Facial Images from Heterogenous Datasets

PurposeTo develop an algorithm to automate the organization of large photo databases using the Haar cascade algorithm for face and eye detection using machine learning tools in Python. DesignRetrospective study for the purposes of clinical tool development. MethodsWe developed an algorithm, termed FaceFinder, to identify front facing images in a large dataset of facial, orthodontal and miscellaneous images. FaceFinder works by detecting the presence of faces and at least two eyes using the Haar cascade. Execution time was recorded using different-sized datasets. A total of 895 images were analyzed by FaceFinder using various thresholds for face and eye detection. Precision, recall, specificity, accuracy, and F1 score were computed by comparison to ground truth labels of the images as determined by a human grader. ResultsUsing medium thresholds for face and eye detection, FaceFinder reached recall, accuracy, and F1 score of 89.3%, 91.6%, and 92.9%, respectively with an execution time per image was 0.91 seconds. Using the highest threshold for face and eye detection, FaceFinder achieved precision and specificity values of 98.3% and 99.2% respectively. ConclusionsFaceFinder is capable of sorting through a heterogenous dataset of photos of patients with craniofacial disease and identifying high-quality front-facing facial images. This capability allows for automated sorting of large databases that can facilitate and expedite data preparation for further downstream analyses involving artificial intelligence and facial landmarking.

Grading of diabetic retinopathy using a pre-segmenting deep learning classification model: Validation of an automated algorithm.

To validate the performance of autonomous diabetic retinopathy (DR) grading by comparing a human grader and a self-developed deep-learning (DL) algorithm with gold-standard evaluation. We included 500, 6-field retinal images graded by an expert ophthalmologist (gold standard) according to the International Clinical Diabetic Retinopathy Disease Severity Scale as represented with DR levels 0-4 (97, 100, 100, 103, 100, respectively). Weighted kappa was calculated to measure the DR classification agreement for (1) a certified human grader without, and (2) with assistance from a DL algorithm and (3) the DL operating autonomously. Using any DR (level 0 vs. 1-4) as a cutoff, we calculated sensitivity, specificity, as well as positive and negative predictive values (PPV and NPV). Finally, we assessed lesion discrepancies between Model 3 and the gold standard. As compared to the gold standard, weighted kappa for Models 1-3 was 0.88, 0.89 and 0.72, sensitivities were 95%, 94% and 78% and specificities were 82%, 84% and 81%. Extrapolating to a real-world DR prevalence of 23.8%, the PPV were 63%, 64% and 57% and the NPV were 98%, 98% and 92%. Discrepancies between the gold standard and Model 3 were mainly incorrect detection of artefacts (n = 49), missed microaneurysms (n = 26) and inconsistencies between the segmentation and classification (n = 51). While the autonomous DL algorithm for DR classification only performed on par with a human grader for some measures in a high-risk population, extrapolations to a real-world population demonstrated an excellent 92% NPV, which could make it clinically feasible to use autonomously to identify non-DR patients.

AI Versus Human Graders: Assessing the Role of Large Language Models in Higher Education

AI Versus Human Graders: Assessing the Role of Large Language Models in Higher Education

Enhancing Faculty Grading Productivity Using Robotic Process Automation (RPA): The Development of a Structured Query Language (SQL) Automated Grading Tool

ABSTRACT Using the design science methodology, we develop an automated grading bot using robotic process automation (RPA) software. This grader demonstrates RPA’s capability to enhance the efficiency and accuracy of grading structured query language (SQL) queries. Validation of the RPA grader demonstrates that it can grade assignments and/or exams in 30 to 60 minutes, a task that would take a human grader approximately 50 hours. In contrast to human graders, who misgraded 5 to 8 percent of queries, the RPA grader graded all queries correctly. Notably, it correctly evaluated 188 distinct correct solutions submitted by students to a single question. To facilitate broad adoption, we provide instructions and the necessary code for implementing the RPA grader. We validate that professors and teaching assistants can successfully implement the RPA grader even with no previous bot building experience.

Telemedicine in Diabetic Retinal Screening: Pre- and Post-COVID-19 Challenges a New Perspective

Abstract Telemedicine in ophthalmology has been around for decades and has been successful with its use in diabetic retinal screening in countries like the UK (with the introduction of the UK National Diabetic Eye Screening Programme in 2003). However, most telemedicine, in the field of diabetic retinopathy, has largely been reliant on human graders for triage purposes. With the advent of COVID-19, patients with chronic conditions, such as diabetes, were disproportionately affected. The pandemic also caused significant rise in patients on waiting lists. Before the pandemic, there have been studies illustrating the use of artificial intelligence (AI) to analyse images obtained from patients screened for monitoring of their diabetic retinopathy. The image analysis by AI and deep-learning algorithms offers insight into the future of screening in diabetes. The transition, from the use of human graders in teleophthalmology to the use of AI-based image analysis has the potential to screen a wider cohort of patients, thereby tackling waiting lists awaiting screening which has lengthened since after COVID-19. It is therefore vital to understand the role of AI in screening diabetic retinopathy patients, from a patient-acceptability, cost-effectiveness and reliability perspective as, this offers potential answers to streamline the screening process further.

Computer Vision Identification of Trachomatous Inflammation-Follicular Using Deep Learning.

Trachoma surveys are used to estimate the prevalence of trachomatous inflammation-follicular (TF) to guide mass antibiotic distribution. These surveys currently rely on human graders, introducing a significant resource burden and potential for human error. This study describes the development and evaluation of machine learning models intended to reduce cost and improve reliability of these surveys. Fifty-six thousand seven hundred twenty-five everted eyelid photographs were obtained from 11,358 children of age 0 to 9 years in a single trachoma-endemic region of Ethiopia over a 3-year period. Expert graders reviewed all images from each examination to determine the estimated number of tarsal conjunctival follicles and the degree of trachomatous inflammation-intense. The median estimate of the 3 grader groups was used as the ground truth to train a MobileNetV3 large deep convolutional neural network to detect cases with TF. The classification model predicted a TF prevalence of 32%, which was not significantly different from the human consensus estimate (30%; 95% confidence interval of difference, -2 to +4%). The model had an area under the receiver operating characteristic curve of 0.943, F1 score of 0.923, 88% accuracy, 83% sensitivity, and 91% specificity. The area under the receiver operating characteristic curve increased to 0.995 when interpreting nonborderline cases of TF. Deep convolutional neural network models performed well at classifying TF and detecting the number of follicles evident in conjunctival photographs. Implementation of similar models may enable accurate, efficient, large-scale trachoma screening. Further validation in diverse populations with varying TF prevalence is needed before implementation at scale.

Capabilities of GPT-4 in ophthalmology: an analysis of model entropy and progress towards human-level medical question answering

BackgroundEvidence on the performance of Generative Pre-trained Transformer 4 (GPT-4), a large language model (LLM), in the ophthalmology question-answering domain is needed.MethodsWe tested GPT-4 on two 260-question multiple choice question...

Grading exams using large language models: A comparison between human and AI grading of exams in higher education using ChatGPT

AbstractThis study compares how the generative AI (GenAI) large language model (LLM) ChatGPT performs in grading university exams compared to human teachers. Aspects investigated include consistency, large discrepancies and length of answer. Implications for higher education, including the role of teachers and ethics, are also discussed. Three Master's‐level exams were scored using ChatGPT 3.5, and the results were compared with the teachers' scoring and the grading teachers were interviewed. In total, 463 exam responses were graded. With each response being graded at least three times, a total of 1389 gradings were conducted. For the final exam scores, 70% of ChatGPT's gradings were within 10% of the teachers' gradings and 31% within 5%. ChatGPT tended to give marginally higher scores. The agreement on grades is 30%, but 45% of the exams received an adjacent grade. On individual questions, ChatGPT is more inclined to avoid very high or very low scores. ChatGPT struggles to correctly score questions closely related to the course lectures but performs better on more general questions. The AI can generate plausible scores on university exams that, at first glance, look similar to a human grader. There are differences but it is not unlikely that two different human graders could result in similar discrepancies. During the interviews, teachers expressed their surprise at how well ChatGPT's grading matched their own. Increased use of AI can lead to ethical challenges as exams are entrusted to a machine whose decision‐making criteria are not fully understood, especially concerning potential bias in training data.

Performance of a Large‐Language Model in scoring construction management capstone design projects

AbstractGrading is one of the most relevant hurdles for instructors, diverting instructor's focus on the development of engaging learning activities, class preparation, and attending to students' questions. Institutions and instructors are continuously looking for alternatives to reduce educators' time required on grading, frequently, resulting in hiring teaching assistants whose inexperience and frequent rotation can lead to inconsistent and subjective evaluations. Large Language Models (LLMs) like GPT‐4 may alleviate grading challenges; however, research in this field is limited when dealing with assignments requiring specialized knowledge, complex critical thinking, subjective, and creative. This research investigates whether GPT‐4's scores correlate with human grading in a construction capstone project and how the use of criteria and rubrics in GPT influences this correlation. Projects were graded by two human graders and three training configurations in GPT‐4: no detailed criteria, paraphrased criteria, and explicit rubrics. Each configuration was tested through 10 iterations to evaluate GPT consistency. Results challenge GPT‐4's potential to grade argumentative assignments. GPT‐4's score correlates slightly better (although poor overall) with human evaluations when no additional information is provided, underscoring the poor impact of the specificity of training materials for GPT scoring in this type of assignment. Despite the LLMs' promises, their limitations include variability in consistency and reliance on statistical pattern analysis, which can lead to misleading evaluations along with privacy concerns when handling sensitive student data. Educators must carefully design grading guidelines to harness the full potential of LLMs in academic assessments, balancing AI's efficiency with the need for nuanced human judgment.

HARNESSING THE POWER OF ARTIFICIAL INTELLIGENCE FOR EARLY DETECTION AND MANAGEMENT OF DIABETIC RETINOPATHY, AGE-RELATED MACULAR DEGENERATION, AND GLAUCOMA: A NARRATIVE REVIEW OF DEEP LEARNING APPLICATIONS IN OPHTHALMOLOGY

Artificial intelligence (AI) and intense learning (DL) models have emerged as powerful tools in ophthalmology, revolutionizing the early detection and management of ocular diseases such as diabetic retinopathy (DR), age-related macular degeneration (AMD), and glaucoma. This narrative review explores AI's current applications and future potential in these domains, focusing on using convolutional neural networks (CNNs) and other DL architectures to analyze retinal fundus photographs, optical coherence tomography (OCT) images, and visual field tests. By leveraging vast datasets and identifying subtle pathological features, AI models have demonstrated high accuracy, sensitivity, and specificity in detecting these diseases, often surpassing human graders. Integrating AI into clinical practice holds promise for enhancing diagnostic efficiency, facilitating early intervention, and ultimately improving patient outcomes. However, challenges related to data quality, model interpretability (the ability to understand and trust the decisions made by AI models), and ethical considerations (such as patient privacy and consent) must be addressed to fully realize AI's potential in ophthalmology. Future research should focus on validating AI models in diverse populations, exploring novel DL architectures, and developing integrated systems seamlessly incorporating AI into clinical workflows.

Automated identification of fleck lesions in Stargardt disease using deep learning enhances lesion detection sensitivity and enables morphometric analysis of flecks

PurposeTo classify fleck lesions and assess artificial intelligence (AI) in identifying flecks in Stargardt disease (STGD).MethodsA retrospective study of 170 eyes from 85 consecutive patients with confirmed STGD. Fundus autofluorescence...

Artificial Intelligence Based Essay Grading System

There are quite a number of challenges faced by teaching staff which if ameliorated can help reduce the quality of time spent on monotonous task associated with students essays grading More so, these days in Nigeria and other related parts of the world institutions deal with a large number of students. These however make grading time consuming and costly, therefore an automated system that can handle the tasks is highly desirable. The developed automated essay grading system in this research is restricted to grading of short and structured essay responses. Java programming language was used for the implementation and MySQL relational database system with WAMPSERVER as the back end. The results showed the user-friendly modules which enable users to navigate through various interfaces easily and work as required. The developed system has an Exact Agreement rate of 0.4 and a Pearson Correlation of 0.93 with human graders.

Automated magnetic resonance imaging-based grading of the lumbar intervertebral disc and facet joints.

Degeneration of both intervertebral discs (IVDs) and facet joints in the lumbar spine has been associated with low back pain, but whether and how IVD/joint degeneration contributes to pain remains an open question. Joint degeneration can be identified by pairing T1 and T2 magnetic resonance imaging (MRI) with analysis techniques such as Pfirrmann grades (IVD degeneration) and Fujiwara scores (facet degeneration). However, these grades are subjective, prompting the need to develop an automated technique to enhance inter-rater reliability. This study introduces an automated convolutional neural network (CNN) technique trained on clinical MRI images of IVD and facet joints obtained from public-access Lumbar Spine MRI Dataset. The primary goal of the automated system is to classify health of lumbar discs and facet joints according to Pfirrmann and Fujiwara grading systems and to enhance inter-rater reliability associated with these grading systems. Performance of the CNN on both the Pfirrmann and Fujiwara scales was measured by comparing the percent agreement, Pearson's correlation and Fleiss kappa value for results from the classifier to the grades assigned by an expert grader. The CNN demonstrates comparable performance to human graders for both Pfirrmann and Fujiwara grading systems, but with larger errors in Fujiwara grading. The CNN improves the reliability of the Pfirrmann system, aligning with previous findings for IVD assessment. The study highlights the potential of using deep learning in classifying the IVD and facet joint health, and due to the high variability in the Fujiwara scoring system, highlights the need for improved imaging and scoring techniques to evaluate facet joint health. All codes required to use the automatic grading routines described herein are available in the Data Repository for University of Minnesota (DRUM).

Screening for diabetic retinopathy with artificial intelligence: a real world evaluation.

Periodic screening for diabetic retinopathy (DR) is effective for preventing blindness. Artificial intelligence (AI) systems could be useful for increasing the screening of DR in diabetic patients. The aim of this study was to compare the performance of the DAIRET system in detecting DR to that of ophthalmologists in a real-world setting. Fundus photography was performed with a nonmydriatic camera in 958 consecutive patients older than 18 years who were affected by diabetes and who were enrolled in the DR screening in the Diabetes and Endocrinology Unit and in the Eye Unit of ULSS8 Berica (Italy) between June 2022 and June 2023. All retinal images were evaluated by DAIRET, which is a machine learning algorithm based on AI. In addition, all the images obtained were analysed by an ophthalmologist who graded the images. The results obtained by DAIRET were compared with those obtained by the ophthalmologist. We included 958 patients, but only 867 (90.5%) patients had retinal images sufficient for evaluation by a human grader. The sensitivity for detecting cases of moderate DR and above was 1 (100%), and the sensitivity for detecting cases of mild DR was 0.84 ± 0.03. The specificity of detecting the absence of DR was lower (0.59 ± 0.04) because of the high number of false-positives. DAIRET showed an optimal sensitivity in detecting all cases of referable DR (moderate DR or above) compared with that of a human grader. On the other hand, the specificity of DAIRET was low because of the high number of false-positives, which limits its cost-effectiveness.

GPT-4 in Education: Evaluating Aptness, Reliability, and Loss of Coherence in Solving Calculus Problems and Grading Submissions

AbstractIn this paper, we initially investigate the capabilities of GPT-3 5 and GPT-4 in solving college-level calculus problems, an essential segment of mathematics that remains under-explored so far. Although improving upon earlier versions, GPT-4 attains approximately 65% accuracy for standard problems and decreases to 20% for competition-like scenarios. Overall, the models prove to be unreliable due to common arithmetic errors.Our primary contribution lies then in examining the use of ChatGPT for grading solutions to calculus exercises. Our objectives are to probe an in-context learning task with less emphasis over direct calculations; recognize positive applications of ChatGPT in educational contexts; highlight a potentially emerging facet of AI that could necessitate oversight; and introduce unconventional AI benchmarks, for which models like GPT are untrained. Pertaining to the latter, we uncover a tendency for loss of coherence in extended contexts. Our findings suggest that while the current ChatGPT exhibits comprehension of the grading task and often provides relevant outputs, the consistency of grading is marred by occasional loss of coherence and hallucinations. Intriguingly, GPT-4's overall scores, delivered in mere moments, align closely with human graders, although its detailed accuracy remains suboptimal.This work suggests that, when appropriately orchestrated, collaboration between human graders and LLMs like GPT-4 might combine their unique strengths while mitigating their respective shortcomings In this direction, it is imperative to consider implementing transparency, fairness, and appropriate regulations in the near future.

Credibility and altered communication styles of AI graders in the classroom

AbstractBackgroundEducation is often the primary arena for exploring and integrating new technologies. AI and human‐machine communication (HMC) are prevalent in the classroom, yet we are still learning how student perceptions of these tools will impact education.ObjectivesWe sought to understand student perceptions of credibility related to written feedback attributed to a human or an AI grader (Study One). We also investigated how corrective messages containing verbal immediacy and social support influenced student perceptions of an AI grader's credibility based on feedback in an evaluated essay (Study Two).MethodsWe used an online experimental design to assess the perceived credibility of a grader. In Study One, we randomly assigned students (N = 155) to a condition that contained a paragraph they were told was evaluated by a human or an AI grader. In Study Two (N = 222), we investigated ways of increasing perceptions of an AI grader's credibility by writing messages with higher/lower levels of immediacy and social support.ResultsIn Study One, the students rated both the human and AI grader as credible (yet rated the AI grader lower on goodwill). The data suggest that students in Study Two attributed more goodwill (i.e., caring) to the AI grader when the feedback included more verbal immediacy.ConclusionsOur results highlight the importance of student perceptions and communication styles when integrating technology into education. The two studies imply that students viewed the human and AI graders as competent, caring, and trustworthy, specifically when feedback included more immediacy cues.

Attention-based deep learning framework for automatic fundus image processing to aid in diabetic retinopathy grading

Background and objectiveEarly detection and grading of Diabetic Retinopathy (DR) is essential to determine an adequate treatment and prevent severe vision loss. However, the manual analysis of fundus images is time consuming and DR screening programs are challenged by the availability of human graders. Current automatic approaches for DR grading attempt the joint detection of all signs at the same time. However, the classification can be optimized if red lesions and bright lesions are independently processed since the task gets divided and simplified. Furthermore, clinicians would greatly benefit from explainable artificial intelligence (XAI) to support the automatic model predictions, especially when the type of lesion is specified. As a novelty, we propose an end-to-end deep learning framework for automatic DR grading (5 severity degrees) based on separating the attention of the dark structures from the bright structures of the retina. As the main contribution, this approach allowed us to generate independent interpretable attention maps for red lesions, such as microaneurysms and hemorrhages, and bright lesions, such as hard exudates, while using image-level labels only. MethodsOur approach is based on a novel attention mechanism which focuses separately on the dark and the bright structures of the retina by performing a previous image decomposition. This mechanism can be seen as a XAI approach which generates independent attention maps for red lesions and bright lesions. The framework includes an image quality assessment stage and deep learning-related techniques, such as data augmentation, transfer learning and fine-tuning. We used the architecture Xception as a feature extractor and the focal loss function to deal with data imbalance. ResultsThe Kaggle DR detection dataset was used for method development and validation. The proposed approach achieved 83.7 % accuracy and a Quadratic Weighted Kappa of 0.78 to classify DR among 5 severity degrees, which outperforms several state-of-the-art approaches. Nevertheless, the main result of this work is the generated attention maps, which reveal the pathological regions on the image distinguishing the red lesions and the bright lesions. These maps provide explainability to the model predictions. ConclusionsOur results suggest that our framework is effective to automatically grade DR. The separate attention approach has proven useful for optimizing the classification. On top of that, the obtained attention maps facilitate visual interpretation for clinicians. Therefore, the proposed method could be a diagnostic aid for the early detection and grading of DR.

Quantification of Choroidal Vascular Hyperpermeability on Ultra-Widefield Indocyanine Green Angiography in Macular Neovascularization.

To obtain a quantitative parameter for the measurement of choroidal vascular hyperpermeability (CVH) on ultra-widefield indocyanine green angiography (UWICGA) using an objective analysis method in macular choroidal neovascularization (CNV). A total of 113 UWICGA images from 113 subjects were obtained, including with 25 neovascular age-related macular degeneration (nAMD), 37 with polypoidal choroidal vasculopathy (PCV) (19 with thin-choroid and 18 with thick-choroid), 33 with pachychoroid neovasculopathy (PNV), and 18 age-matched controls. CVH was quantified on a gray image by the subtraction of 2 synchronized UWICGA images of early and late phases. The measured CVH parameter was compared with human graders and among CNV subtypes and correlated with choroidal vascular density (CVD) and subfoveal choroidal thickness (SFCT). The mean CVH values were 28.58 ± 4.97, 33.36 ± 8.40, 33.61 ± 11.50, 42.19 ± 13.25, and 43.59 ± 7.86 in controls and patients with nAMD, thin-choroid PCV, thick-choroid PCV, and PNV, respectively (p < 0.001). CVH was higher in thick-choroid PCV and PNV compared to the other groups (all p ≤ 0.006). The measured CVH value positively correlated with those reported by human graders (p < 0.001), CVD, and SFCT (p = 0.001 and p < 0.001, respectively). CVH can be measured objectively using quantitative UWICGA analysis. The CVH parameter differs among macular CNV subtypes and correlates with CVD and SFCT.

DeBERTa with hats makes Automated Essay Scoring system better

Automated Essay Scoring (AES) is a rapidly growing field that applies natural language processing (NLP) and machine learning techniques to the analysis and evaluation of academic essays. By automating the process of evaluating essay quality, AES not only greatly reduces the workload of human graders but also ensures consistency and objectivity in the evaluation process. AES systems can evaluate essays based on multiple criteria, including organization, coherence, and content. With the advent of deep learning, AES has shown significant improvements in accuracy and reliability. AES systems have numerous applications in education, particularly in large-scale assessment and feedback loops. In this article, we delve into the use of an improved Bidirectional Encoder Representations from Transformers (BERT) architecture with disentangled attention mechanism known as DeBERTa for student question-based summarization. This is one of the downstream tasks within AES, which is of great significance for student learning assessment. The organic combination of DeBERTa-v3 and diverse hats like Light Gradient Boosting Machine (LGBM) algorithm and Extreme Gradient Boosting algorithm (XGBoost) has proven to be highly effective in achieving excellent results in this task, indicating their significant potential in real-world AES systems.