Human Evaluation Research Articles

237 SeeMe: An Automated Methodology to Detect Early Signs of Recovery After Acute Brain Injury

INTRODUCTION: Assessing the functional state of the brain and predicting recovery after acute brain injury (ABI) remains a major unsolved problem. At present, physicians make predictions using clinical examination, traditional scoring systems, and statistical models, which have significant shortcomings and lack the ability to forecast recovery on an individual basis. METHODS: SeeMe is based on analyzing subtle facial expressions in response to auditory commands (i.e., “open your eyes,” “make a smile”) over time and compares the results to a pre-stimulus baseline. These subtle motions are often not detectable by the naked eye. We enrolled 26 comatose ABI patients and 16 healthy volunteers to test this tool. Command-evoked eye-opening movements were compared to clinical observation and the JFK Coma Recovery Scale-Revised (CRS-R). RESULTS: In our study, 22 of 26 (85%) patients recovered consciousness during hospitalization after ABI. On average, SeeMe detected command-evoked eye opening ∼9 days before clinical assessment using the CRS-R and ∼3 days before blinded human raters who observed the recorded videos using the naked eye.Using the optimally identified decision threshold, SeeMe demonstrated the following performance metrics on ABI patients with no eye opening to command (auditory CRS subscale = 2): sensitivity = 74.1%, specificity = 66.6%, positive predictive value = 16.8%, and negative predictive value = 96.6%. CONCLUSIONS: These findings demonstrate the high predictive performance of SeeMe in assessing consciousness in ABI patients and provide a quantitative tool for studying neural circuits and developing novel therapeutic approaches. SeeMe's ability to detect low-amplitude motor movements opens new avenues for understanding and facilitating recovery of consciousness in patients with ABI.

Formative Feedback on Student-Authored Summaries in Intelligent Textbooks Using Large Language Models

AbstractAs intelligent textbooks become more ubiquitous in classrooms and educational settings, the need to make them more interactive arises. An alternative is to ask students to generate knowledge in response to textbook content and provide feedback about the produced knowledge. This study develops Natural Language Processing models to automatically provide feedback to students about the quality of summaries written at the end of intelligent textbook sections. The study builds on the work of Botarleanu et al. (2022), who used a Longformer Large Language Model (LLM) to develop a summary grading model. Their model explained around 55% of holistic summary score variance as assigned by human raters. This study uses a principal component analysis to distill summary scores from an analytic rubric into two principal components – content and wording. This study uses two encoder-only classification large language models finetuned from Longformer on the summaries and the source texts using these principal components explained 82% and 70% of the score variance for content and wording, respectively. On a dataset of summaries collected on the crowd-sourcing site Prolific, the content model was shown to be robust although the accuracy of the wording model was reduced compared to the training set. The developed models are freely available on HuggingFace and will allow formative feedback to users of intelligent textbooks to assess reading comprehension through summarization in real time. The models can also be used for other summarization applications in learning systems.

Methodological insights into ChatGPT’s screening performance in systematic reviews

BackgroundThe screening process for systematic reviews and meta-analyses in medical research is a labor-intensive and time-consuming task. While machine learning and deep learning have been applied to facilitate this process, these methods often require training data and user annotation. This study aims to assess the efficacy of ChatGPT, a large language model based on the Generative Pretrained Transformers (GPT) architecture, in automating the screening process for systematic reviews in radiology without the need for training data.MethodsA prospective simulation study was conducted between May 2nd and 24th, 2023, comparing ChatGPT’s performance in screening abstracts against that of general physicians (GPs). A total of 1198 abstracts across three subfields of radiology were evaluated. Metrics such as sensitivity, specificity, positive and negative predictive values (PPV and NPV), workload saving, and others were employed. Statistical analyses included the Kappa coefficient for inter-rater agreement, ROC curve plotting, AUC calculation, and bootstrapping for p-values and confidence intervals.ResultsChatGPT completed the screening process within an hour, while GPs took an average of 7–10 days. The AI model achieved a sensitivity of 95% and an NPV of 99%, slightly outperforming the GPs’ sensitive consensus (i.e., including records if at least one person includes them). It also exhibited remarkably low false negative counts and high workload savings, ranging from 40 to 83%. However, ChatGPT had lower specificity and PPV compared to human raters. The average Kappa agreement between ChatGPT and other raters was 0.27.ConclusionsChatGPT shows promise in automating the article screening phase of systematic reviews, achieving high sensitivity and workload savings. While not entirely replacing human expertise, it could serve as an efficient first-line screening tool, particularly in reducing the burden on human resources. Further studies are needed to fine-tune its capabilities and validate its utility across different medical subfields.

Using machine learning for continuous updating of meta-analysis in educational context

Machine learning and learning analytics are powerful tools that not only support researchers in the detailed measurement and enhancement of learning processes in various learning environments, but also enable the aggregation and synthesis of evidence regarding effective educational practices. This paper describes the development and application of machine learning algorithms aimed at semi-automatic selection of abstracts for a meta-analysis on the effects of simulation-based learning in higher education. The goal was to reduce the workload while also maintaining the transparency and objectivity of the selection process. The algorithms were trained, validated, and tested on a set of 3187 studies on simulation-based learning found in medical and educational databases collected before April 2018. Subsequently, they were utilized to classify abstracts for a follow-up meta-analysis consisting of 2373 studies (published between 2018 and 2020). The aim of training the algorithms was to predict studies’ abstract eligibility based on words and combinations of words used in these abstracts. The application of the algorithms reduced the number of studies that had to be manually screened from 2373 to 711. A total of 458 studies from automatically selected abstracts were included in the full-text screening, indicating the high precision of the algorithms (also compared to the performance of human raters). We conclude that machine learning algorithms can be trained and used to classify abstracts for their eligibility, significantly reducing the workload for the researchers without diminishing objectivity and quality when updating systematic literature reviews with or without a meta-analysis.

Capuchin monkeys' (Sapajus [Cebus] apella) categorization of photos of unknown male conspecifics suggests attention to fWHR and a dominance bias.

The ability to quickly perceive others' rank minimizes costs by helping individuals behave appropriately when interacting with strangers. Indeed, humans and at least some other species can quickly determine strangers' rank or dominance based only on physical features without observing others' interactions or behavior. Nonhuman primates can determine strangers' ranks by observing their interactions, and some evidence suggests that at least some cues to dominance, such as facial width-to-height ratio (fWHR), are also present in other primates. However, it is unknown whether they can determine strangers' rank simply by looking at their faces, rather than observing their interactions. If so, this would suggest selective pressure across the primates on both cues to dominance and the ability to detect those cues accurately. To address this, we examined the ability of male and female tufted capuchin monkeys (Sapajus [Cebus] apella) to categorize images of the faces of unknown conspecifics (Sapajus from different colonies) and humans (computer-generated and real) as dominant or nondominant based only on still images. Capuchins' categorization of unknown conspecific faces was consistent with fWHR, a cue to dominance, although there was a strong tendency to categorize strangers as dominant, particularly for males. This was true despite the continued correct categorization of known individuals. In addition, capuchins did not categorize human strangers in accordance with external pre-ratings of dominance by independent human raters, despite the availability of the same cue, fWHR. We consider these results in the context of capuchin socio-ecology and what they mean for the evolution of rapid decision-making in social contexts.

Automatic Scoring of Metaphor Creativity with Large Language Models

ABSTRACT Metaphor is crucial in human cognition and creativity, facilitating abstract thinking, analogical reasoning, and idea generation. Typically, human raters manually score the originality of responses to creative thinking tasks – a laborious and error-prone process. Previous research sought to remedy these risks by scoring creativity tasks automatically using semantic distance and large language models (LLMs). Here, we extend research on automatic creativity scoring to metaphor generation – the ability to creatively describe episodes and concepts using nonliteral language. Metaphor is arguably more abstract and naturalistic than prior targets of automated creativity assessment. We collected 4,589 responses from 1,546 participants to various metaphor prompts and corresponding human creativity ratings. We fine-tuned two open-source LLMs (RoBERTa and GPT-2) – effectively “teaching” them to score metaphors like humans – before testing their ability to accurately assess the creativity of new metaphors. Results showed both models reliably predicted new human creativity ratings (RoBERTa r = .72, GPT-2 r = .70), significantly more strongly than semantic distance (r = .42). Importantly, the fine-tuned models generalized accurately to metaphor prompts they had not been trained on (RoBERTa r = .68, GPT-2 r = .63). We provide open access to the fine-tuned models, allowing researchers to assess metaphor creativity in a reproducible and timely manner.

Facial recognition technology and human raters can predict political orientation from images of expressionless faces even when controlling for demographics and self-presentation.

Carefully standardized facial images of 591 participants were taken in the laboratory while controlling for self-presentation, facial expression, head orientation, and image properties. They were presented to human raters and a facial recognition algorithm: both humans (r = .21) and the algorithm (r = .22) could predict participants' scores on a political orientation scale (Cronbach's α = .94) decorrelated with age, gender, and ethnicity. These effects are on par with how well job interviews predict job success, or alcohol drives aggressiveness. The algorithm's predictive accuracy was even higher (r = .31) when it leveraged information on participants' age, gender, and ethnicity. Moreover, the associations between facial appearance and political orientation seem to generalize beyond our sample: The predictive model derived from standardized images (while controlling for age, gender, and ethnicity) could predict political orientation (r ≈ .13) from naturalistic images of 3,401 politicians from the United States, the United Kingdom, and Canada. The analysis of facial features associated with political orientation revealed that conservatives tended to have larger lower faces. The predictability of political orientation from standardized images has critical implications for privacy, the regulation of facial recognition technology, and understanding the origins and consequences of political orientation. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Large language models know how the personality of public figures is perceived by the general public

We show that people’s perceptions of public figures’ personalities can be accurately predicted from their names’ location in GPT-3’s semantic space. We collected Big Five personality perceptions of 226 public figures from 600 human raters. Cross-validated linear regression was used to predict human perceptions from public figures’ name embeddings extracted from GPT-3. The models’ accuracy ranged from r = .78 to .88 without controls and from r = .53 to .70 when controlling for public figures’ likability and demographics, after correcting for attenuation. Prediction models showed high face validity as revealed by the personality-descriptive adjectives occupying their extremes. Our findings reveal that GPT-3 word embeddings capture signals pertaining to individual differences and intimate traits.

Raters’ scoring process in assessment of interpreting: an empirical study based on eye tracking and retrospective verbalisation

ABSTRACT Human raters’ assessment of interpreting is a complex process. Previous researchers have mainly relied on verbal reports to examine this process. To advance our understanding, we conducted an empirical study, collecting raters’ eye-movement and retrospection data in a computerised interpreting assessment in which three groups of raters (n = 35) used an analytic rubric to assess 12 English-to-Chinese consecutive interpretations. We examined how the raters interacted with the source text, the rating scale, and the audio player displayed on the computer screen when they were assessing. We found that a) the source text and the rating scale were competing for the raters’ visual attention, with the former attracting more attention than the latter across the rater groups; b) when the raters were consulting the rating scale, they fixated less frequently on the sub-scale of target language quality than the other two sub-scales; c) the rater groups did not seem to exhibit substantially discrepant gazing behaviours overall, although there emerged different eye-movement patterns concerning certain sub-scales; and d) the raters utilised an array of strategies and shortcuts to facilitate their assessment. We discuss these findings in relation to rater training and validation of score meaning for interpreting assessment.

Automatic hepatic tumor segmentation in intra-operative ultrasound: a supervised deep-learning approach.

Training and evaluation of the performance of a supervised deep-learning model for the segmentation of hepatic tumors from intraoperative US (iUS) images, with the purpose of improving the accuracy of tumor margin assessment during liver surgeries and the detection of lesions during colorectal surgeries. In this retrospective study, a U-Net network was trained with the nnU-Net framework in different configurations for the segmentation of CRLM from iUS. The model was trained on B-mode intraoperative hepatic US images, hand-labeled by an expert clinician. The model was tested on an independent set of similar images. The average age of the study population was 61.9 ± 9.9 years. Ground truth for the test set was provided by a radiologist, and three extra delineation sets were used for the computation of inter-observer variability. The presented model achieved a DSC of 0.84 (), which is comparable to the expert human raters scores. The model segmented hypoechoic and mixed lesions more accurately (DSC of 0.89 and 0.88, respectively) than hyper- and isoechoic ones (DSC of 0.70 and 0.60, respectively) only missing isoechoic or >20mm in diameter (8% of the tumors) lesions. The inclusion of extra margins of probable tumor tissue around the lesions in the training ground truth resulted in lower DSCs of 0.75 (). The model can accurately segment hepatic tumors from iUS images and has the potential to speed up the resection margin definition during surgeries and the detection of lesion in screenings by automating iUS assessment.

Enhancing Rubric Development in Science Education through Topic Modeling Techniques

Content experts conduct traditional approaches to developing rubrics for constructed response (CR) items, which is a labor-intensive process. We aim to illustrate the potential benefits of utilizing topic modeling techniques to improve the efficiency of the rubric workload. The results of this study reveal that over half of the keywords in the rubric corresponded with the top 20 words extracted from the supervised latent Dirichlet allocation (sLDA) model. The rubric-based scores assigned by human raters showed medium-to-high associations with the sLDA-predicted scores. The sLDA approach offers supporting evidence for refining the rubric concerning criteria for word usage and score assignments. It accomplishes this by concurrently incorporating the predetermined rubric of content experts and the actual responses provided by students.

Using natural language processing to increase prediction and reduce subgroup differences in personnel selection decisions.

The purpose of this research is to demonstrate how using natural language processing (NLP) on narrative application data can improve prediction and reduce racial subgroup differences in scores used for selection decisions compared to mental ability test scores and numeric application data. We posit there is uncaptured and job-related constructs that can be gleaned from applicant text data using NLP. We test our hypotheses in an operational context across four samples (total N = 1,828) to predict selection into Officer Training School in the U.S. Air Force. Boards of three senior officers make selection decisions using a highly structured rating process based on mental ability tests, numeric application information (e.g., number of past jobs, college grades), and narrative application information (e.g., past job duties, achievements, interests, statements of objectives). Results showed that NLP scores of the narrative application generally (a) predict Board scores when combined with test scores and numeric application information at a level of correlation equivalent to the correlation between human raters (.60), (b) add incremental prediction of Board scores beyond mental ability tests and numeric application information, and (c) reduce subgroup differences between racial minorities and nonracial minorities in Board scores compared to mental ability tests and numeric application information. Moreover, NLP scores predict (a) job (training) performance, (b) job (training) performance beyond mental ability tests and numeric application information, and (c) even job (training) performance beyond Board scores. Scoring of narrative application data using NLP shows promise in addressing the validity-adverse impact dilemma in selection. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Automatic Scoring of Rhizoctonia Crown and Root Rot Affected Sugar Beet Fields from Orthorectified UAV Images Using Machine Learning.

Rhizoctonia crown and root rot (RCRR), caused by Rhizoctonia solani, can cause severe yield and quality losses in sugar beet. The most common strategy to control the disease is the development of resistant varieties. In the breeding process, field experiments with artificial inoculation are carried out to evaluate the performance of genotypes and varieties. The phenotyping process in breeding trials requires constant monitoring and scoring by skilled experts. This work is time demanding and shows bias and heterogeneity according to the experience and capacity of each individual person. Optical sensors and artificial intelligence have demonstrated great potential to achieve higher accuracy than human raters and the possibility to standardize phenotyping applications. A workflow combining red-green-blue and multispectral imagery coupled to an unmanned aerial vehicle (UAV), as well as machine learning techniques, was applied to score diseased plants and plots affected by RCRR. Georeferenced annotation of UAV-orthorectified images was carried out. With the annotated images, five convolutional neural networks were trained to score individual plants. The training was carried out with different image analysis strategies and data augmentation. The custom convolutional neural network trained from scratch together with pretrained MobileNet showed the best precision in scoring RCRR (0.73 to 0.85). The average per plot of spectral information was used to score the plots, and the benefit of adding the information obtained from the score of individual plants was compared. For this purpose, machine learning models were trained together with data management strategies, and the best-performing model was chosen. A combined pipeline of random forest and k-nearest neighbors has shown the best weighted precision (0.67). This research provides a reliable workflow for detecting and scoring RCRR based on aerial imagery. RCRR is often distributed heterogeneously in trial plots; therefore, considering the information from individual plants of the plots showed a significant improvement in UAV-based automated monitoring routines.

A Comparative Analysis of the Rating of College Students’ Essays by ChatGPT versus Human Raters

The use of generative artificial intelligence (AI) in education has engendered mixed reactions due to its ability to generate human-like responses to questions. For education to benefit from this modern technology, there is a need to determine how such capability can be used to improve teaching and learning. Hence, using a comparative−descriptive research design, this study aimed to perform a comparative analysis between Chat Generative Pre-Trained Transformer (ChatGPT) version 3.5 and human raters in scoring students’ essays. Twenty essays were used of college students in a professional education course at the Mindanao State University – Tawi-Tawi College of Technology and Oceanography, a public university in southern Philippines. The essays were rated independently by three human raters using a scoring rubric from Carrol and West (1989) as adapted by Tuyen et al. (2019). For the AI ratings, the essays were encoded and inputted into ChatGPT 3.5 using prompts and the rubric. The responses were then screenshotted and recorded along with the human ratings for statistical analysis. Using the intraclass correlation coefficient (ICC), results show that among the human raters, the consistency was good, indicating the reliability of the rubric, while a moderate consistency was found in the ChatGPT 3.5 ratings. Comparison of the human and ChatGPT 3.5 ratings show poor consistency, implying the that the ratings of human raters and ChatGPT 3.5 were not linearly related. The finding implies that teachers should be cautious when using ChatGPT in rating students’ written works, suggesting further that using ChatGPT 3.5, in its current version, still needs human assistance to ensure the accuracy of its generated information. Rating of other types of student works using ChatGPT 3.5 or other generative AI tools may be investigated in future research.

Disentangled deep generative models reveal coding principles of the human face processing network.

Despite decades of research, much is still unknown about the computations carried out in the human face processing network. Recently, deep networks have been proposed as a computational account of human visual processing, but while they provide a good match to neural data throughout visual cortex, they lack interpretability. We introduce a method for interpreting brain activity using a new class of deep generative models, disentangled representation learning models, which learn a low-dimensional latent space that "disentangles" different semantically meaningful dimensions of faces, such as rotation, lighting, or hairstyle, in an unsupervised manner by enforcing statistical independence between dimensions. We find that the majority of our model's learned latent dimensions are interpretable by human raters. Further, these latent dimensions serve as a good encoding model for human fMRI data. We next investigate the representation of different latent dimensions across face-selective voxels. We find that low- and high-level face features are represented in posterior and anterior face-selective regions, respectively, corroborating prior models of human face recognition. Interestingly, though, we find identity-relevant and irrelevant face features across the face processing network. Finally, we provide new insight into the few "entangled" (uninterpretable) dimensions in our model by showing that they match responses in the ventral stream and carry information about facial identity. Disentangled face encoding models provide an exciting alternative to standard "black box" deep learning approaches for modeling and interpreting human brain data.

Offering an olive branch: a study of dissenting rater's practices for resolving placement discrepancies

Assessment of student writing by human raters often involves discrepancies, necessitating discussions for resolution. This study explores this process within a US university's English for Academic Purposes program, focusing on the exit practices of dissenting raters - strategies used to conclude disagreements. Utilizing multimodal conversation analysis, the study reveals that dissenting raters typically employ two exit practices: acknowledging personal bias and deferring to the opinions of co-present raters. While acknowledging bias effectively ends disagreements, deferring to others can be perceived as challenging equal participation, thus introducing complexity into the resolution process. The study also highlights how social concerns regarding professional credibility and reputation influence raters’ resolution strategies. The findings offer valuable insights into the interactional complexities of academic placement meetings, underscoring the importance of understanding exit practices in these contexts. They also demonstrate the potential of a conversation analysis approach for uncovering unexplored facets of rater interactions.

Artificial Intelligence Versus Human Focus Group Rating of Facial Attractiveness.

Background: Many open-access artificial intelligence (AI)-based websites that rate facial attractiveness are available, but none have been compared with human focus group outcomes. Objective: To compare human and AI-based websites scoring of facial attractiveness of adult female white faces. Methods: A 40-photograph database of AI-generated adult, white, female, expressionless, and frontal-view facial images were scored by otolaryngology residents and five AI-based facial rating websites: prettyscale.com, attractivenesstest.com, face-score.com/en, hotchat3000.com, and beautyscoretest.com. Sample t-test and bivariate correlation were performed for statistical analyses. Results: The focus group of 24 otolaryngology residents consisted of 62.5% males and 58.3% white participants. There was a strong positive correlation between average human score and average AI score for each photo (Pearson's correlation 0.84, p < 0.01). The average human raters' scores were significantly lower than the average AI scores (5.0 ± 1.8 vs. 6.9 ± 0.9, p < 0.01). Thirty images (75.0%) had statistically higher scores from the AI websites versus the focus group. On correlation analysis, all AI-based websites individually had scores that positively correlate with the human scores (all p < 0.05). Conclusion: AI-based websites and human focus-group scoring of facial attractiveness of adult white female faces were significantly correlated with the AI ratings biased toward higher values, encouraging their cautious utilization in future research.

A Comparison of Head Movement Classification Methods.

To understand human behavior, it is essential to study it in the context of natural movement in immersive, three-dimensional environments. Virtual reality (VR), with head-mounted displays, offers an unprecedented compromise between ecological validity and experimental control. However, such technological advancements mean that new data streams will become more widely available, and therefore, a need arises to standardize methodologies by which these streams are analyzed. One such data stream is that of head position and rotation tracking, now made easily available from head-mounted systems. The current study presents five candidate algorithms of varying complexity for classifying head movements. Each algorithm is compared against human rater classifications and graded based on the overall agreement as well as biases in metrics such as movement onset/offset time and movement amplitude. Finally, we conclude this article by offering recommendations for the best practices and considerations for VR researchers looking to incorporate head movement analysis in their future studies.

(021) ChatGPT's Ability to Assess Quality and Readability of Online Medical Information

Abstract Introduction Health literacy plays a crucial role in enabling patients to understand and effectively use medical information. As technology rapidly advances, the significance of health literacy becomes even more pronounced, particularly in comprehending complex medical information. Artificial Intelligence (AI) platforms have garnered significant attention for their remarkable ability to generate automated responses to a wide range of prompts. However, their capacity to assess the quality and readability of provided text remains uncertain. Given the growing prominence of AI web assistant tools, we hypothesized that integrating these tools into patients' web searches could enhance the retrieval of accurate medical information. Objective To evaluate the proficiency of Conversational Generative Pre-Trained Transformer (ChatGPT) in assessing readability, and utilizing the DISCERN tool to assess quality of online content regarding shock wave therapy for erectile dysfunction. Methods Websites were generated using a Google search of “shock wave therapy for erectile dysfunction” with location filters disabled. Readability was analyzed using the Readable software (Readable.com, Horsham, United Kingdom). Quality was assessed independently by three reviewers using the DISCERN tool. The same plain text files collected were inputted into ChatGPT to determine whether it produced comparable metrics for readability and quality. Results The study results revealed a notable disparity between ChatGPT's readability assessment and that obtained from a reliable tool, Readable.com (p&amp;lt;0.05). This indicates a lack of alignment between ChatGPT's algorithm and that of established tools, such as Readable.com. Similarly, the DISCERN score generated by ChatGPT differed significantly from the scores generated manually by human evaluators (p&amp;lt;0.05), suggesting that ChatGPT may not be capable of accurately identifying poor-quality information sources regarding shock wave therapy as a treatment for erectile dysfunction. Conclusions ChatGPT’s evaluation of the quality and readability of online text regarding shockwave therapy for erectile dysfunction differs from that of human raters and trusted tools. ChatGPT's current capabilities were not sufficient for reliably assessing the quality and readability of textual content. Further research is needed to elucidate the role of AI in the objective evaluation of online medical content in other fields. Continued development in AI and incorporation of tools such as DISCERN into AI software may enhance the way patients navigate the web in search of high-quality medical content in the future. Disclosure No.

Using Fleiss’ Kappa Coefficient to Measure the Intra and Inter-Rater Reliability of Three AI Software Programs in the Assessment of EFL Learners’ Story Writing

Story writing is a valuable skill for EFL learners, as it allows them to express their creativity and practice their language proficiency. However, assessing story writing can be challenging and time-consuming for teachers, especially when they have to deal with large classes and multiple criteria. Therefore, some researchers have explored the use of artificial intelligence (AI) tools to automate the assessment of story writing and provide feedback to learners. However, the reliability of these tools is still questionable. This study aimed to compare the intra- and inter-rater reliability of three AI tools for assessing EFL learners' story writing: Poe.com, Bing, and Google Bard. The study utilized quantitative tools to answer the research questions, namely, calculating the Fleiss' Kappa coefficient using the Datatab software program (available on datatab.com). The study sampled 14 written pieces by EFL Libyan adult learners, the pieces used were stories built around a prompt provided by the teacher. The assessment was done using two criteria, one including the measurement of students' creativity, and the second was done focusing only on the linguistic aspect of the students' writings. With the creativity criterion, the results of the study show that Poe's intra-rater reliability was 0.01 (slight), while Bing's was 0.2 (fair), Bard's was 0.2 (fair). This shows that Poe is the least reliable assessment tool among the three. For the inter-rater reliability, there were three assessments done to the same 14 sampled pieces to check the consistency of the results. In the first attempt the inter-rater reliability was 0.04 (slight), the second assessment it was 0.01 (slight), on the third time it was -0.03 (no agreement). There was a decrease in the consistency and reliability of scores over time. Without the creativity criterion, the results show that Poe's inter-rater reliability level was 0.05 (slight), while Bing's was -0.02 (no agreement), and Bard's was 0.01 (slight). Here, it is shown that Bing was the least reliable. For the inter-rater reliability, the three assessments made by the three software applications were compared. There were three assessments done on the same 14 sampled pieces to check the consistency of the results. In the first attempt, the inter-rater reliability was 0 (slight), the second assessment it was -0.1 (no agreement), on the third time it was -0.13 (no agreement). There was a decrease in the consistency and reliability of scores over time. The three applications performed in a reliable way to a certain extent without the exclusion of the creativity criterion, this goes against the common belief that AI software cannot assess creativity. Still, the results of the reliability measurements with the creativity criterion show that the assessment scores are not statistically significant, and there's a high probability that the observed agreement is due to random chance. Some limitations of this study were the small sample size, the limited number of criteria, and the lack of human raters for comparison. Future research could involve more participants, more criteria, more AI tools, and human raters to provide a more comprehensive and reliable evaluation of AI tools for assessing EFL story writing.