Human Judgment Research Articles

Evaluating AI and human authorship quality in academic writing through physics essays

This study aims to compare the academic writing quality and detectability of authorship between human and AI-generated texts by evaluating n = 300 short-form physics essay submissions, equally divided between student work submitted before the introduction of ChatGPT and those generated by OpenAI’s GPT-4. In blinded evaluations conducted by five independent markers who were unaware of the origin of the essays, we observed no statistically significant differences in scores between essays authored by humans and those produced by AI (p-value = 0.107, α = 0.05). Additionally, when the markers subsequently attempted to identify the authorship of the essays on a 4-point Likert scale—from ‘Definitely AI’ to ‘Definitely Human’—their performance was only marginally better than random chance. This outcome not only underscores the convergence of AI and human authorship quality but also highlights the difficulty of discerning AI-generated content solely through human judgment. Furthermore, the effectiveness of five commercially available software tools for identifying essay authorship was evaluated. Among these, ZeroGPT was the most accurate, achieving a 98% accuracy rate and a precision score of 1.0 when its classifications were reduced to binary outcomes. This result is a source of potential optimism for maintaining assessment integrity. Finally, we propose that texts with ≤50% AI-generated content should be considered the upper limit for classification as human-authored, a boundary inclusive of a future with ubiquitous AI assistance whilst also respecting human-authorship.

The effect of calibration training on the calibration of intelligence analysts' judgments

AbstractExperts are expected to make well‐calibrated judgments within their field, yet a voluminous literature demonstrates miscalibration in human judgment. Calibration training aimed at improving subsequent calibration performance offers a potential solution. We tested the effect of commercial calibration training on a group of 70 intelligence analysts by comparing the miscalibration and bias of their judgments before and after a commercial training course meant to improve calibration across interval estimation and binary choice tasks. Training significantly improved calibration and bias overall, but this effect was contingent on the task. For interval estimation, analysts were overconfident before training and became better calibrated after training. For the binary choice task, however, analysts were initially underconfident and bias increased in this same direction post‐training. Improvement on the two tasks was also uncorrelated. Taken together, results indicate that the training shifted analyst bias toward less confidence rather than having improved metacognitive monitoring ability.

SenseMap: Urban Performance Visualization and Analytics Via Semantic Textual Similarity.

As urban populations grow, effectively accessing urban performance measures such as livability and comfort becomes increasingly important due to their significant socioeconomic impacts. While Point of Interest (POI) data has been utilized for various applications in location-based services, its potential for urban performance analytics remains unexplored. In this article, we present SenseMap, a novel approach for analyzing urban performance by leveraging POI data as a semantic representation of urban functions. We quantify the contribution of POIs to different urban performance measures by calculating semantic textual similarities on our constructed corpus. We propose Semantic-adaptive Kernel Density Estimation which takes into account POIs' influential areas across different Traffic Analysis Zones and semantic contributions to generate semantic density maps for measures. We design and implement a feature-rich, real-time visual analytics system for users to explore the urban performance of their surroundings. Evaluations with human judgment and reference data demonstrate the feasibility and validity of our method. Usage scenarios and user studies demonstrate the capability, usability and explainability of our system.

Studying the Effect of Material and Geometry on Perceptual Outdoor Illumination.

Understanding and modeling perceived properties of sky-dome illumination is an important but challenging problem due to the interplay of several factors such as the materials and geometries of the objects present in the scene being observed. Existing models of sky-dome illumination focus on the physical properties of the sky. However, these parametric models often do not align well with the properties perceived by a human observer. In this work, drawing inspiration from the Hosek-Wilkie sky-dome model, we investigate the perceptual properties of outdoor illumination. For this purpose, we perform a large-scale user study via crowdsourcing to collect a dataset of perceived illumination properties (scattering, glare, and brightness) for different combinations of geometries and materials under a variety of outdoor illuminations, totaling 5,000 distinct images. We perform a thorough statistical analysis of the collected data which reveals several interesting effects. For instance, our analysis shows that when there are objects in the scene made of rough materials, the perceived scattering of the sky increases. Furthermore, we utilize our extensive collection of images and their corresponding perceptual attributes to train a predictor. This predictor, when provided with a single image as input, generates an estimation of perceived illumination properties that align with human perceptual judgments. Accurately estimating perceived illumination properties can greatly enhance the overall quality of integrating virtual objects into real scene photographs. Consequently, we showcase various applications of our predictor. For instance, we demonstrate its utility as a luminance editing tool for showcasing virtual objects in outdoor scenes.

Blinded by the Data: When Analytics Overshadows Judgment

This article discusses the potential risks that can emerge when organizations overly on data-driven decision making by focusing excessively on quantitative analysis and metrics to the exclusion of important qualitative human factors. The article explores common pitfalls such as leaders losing sight of broader strategic perspectives and higher-level business drivers when immersed in granular data points, organizational chaos and paralysis resulting from constant A/B testing without clear objectives, over-quantification discounting important but difficult to measure "soft" elements like culture, talent and leadership that profoundly shape performance, and overconfidence in the perceived objectivity of models overlooks inherent biases. The article advocates balancing quantitative and qualitative lenses to gain a more textured understanding of complex dynamics by supplementing metrics with perspectives from ethnography and stakeholder input to provide needed context as numbers alone do not replace prudent application of human judgment and experience, and for data-driven initiatives to achieve tangible impacts, analytics must inform innovation and change management rather than replace continuous improvement processes, such that with moderation and integration of multiple viewpoints, data can empower rather than control strategic decision making.

The Relationship between Human Rights Judgment of Pre-service Primary School Teachers and Taking Human Rights Education Courses at Universities

The Relationship between Human Rights Judgment of Pre-service Primary School Teachers and Taking Human Rights Education Courses at Universities

Heuristics and biases in human–algorithm interaction and hotel revenue management override decision-making

Purpose This study aims to provide insights into human–algorithm interaction in revenue management (RM) decision-making and to uncover the underlying heuristics and biases of overriding systems’ recommendations. Design/methodology/approach Following constructivist traditions, 20 in-depth interviews were conducted with revenue optimisers, analysts, managers and directors with vast experience in over 25 markets and working with different RM systems (RMSs) at the property and corporate levels. The hermeneutics approach was used to interpret and make meaning of the participants’ lived experiences and interactions with RMSs. Findings The findings explain the nature of the interaction between RM professionals and RMSs, the cognitive mechanism by which the system users judgementally adjust or override its recommendations and the heuristics and biases behind override decisions. Additionally, the findings reveal the individual decision-maker characteristics and organisational factors influencing human–algorithm interactions. Research limitations/implications Although the study focused on human–system interaction in hotel RM, it has larger implications for integrating human judgement into computerised systems for optimal decision-making. Practical implications The study findings expose human biases in working with RMSs and highlight the influencing factors that can be addressed to achieve effective human–algorithm interactions. Originality/value The study offers a holistic framework underpinned by the organisational role and expectation confirmation theories to explain the cognitive mechanisms of human–system interaction in managerial decision-making.

Advanced Fuzzy-Based Decision-Making: The Linear Diophantine Fuzzy CODAS Method for Logistic Specialist Selection

This study addresses the inherent uncertainty in human decision-making by leveraging fuzzy sets, which were introduced to better capture the imprecision associated with human thoughts and judgments. As an extension to traditional fuzzy sets, the Linear Diophantine Fuzzy Set (LDFS) was developed, offering a more flexible approach by relaxing existing limitations on grade values. The LDFS has found applications across various fields, demonstrating its versatility and effectiveness. In this study, we explore the application of the Linear Diophantine Fuzzy Set within the framework of the Combinative Distance-based Assessment (CODAS) method. The CODAS method stands out because it incorporates Euclidean and Taxicab distances. Decision-making should consider not only the direct distances between ideal solutions and alternatives but also the indirect distances. The foremost objective of this research is to propose a novel approach by integrating the CODAS method with the LDFS to address complex decision-making problems characterized by uncertainty and imprecision. To illustrate the practical utility of the proposed method, we apply it to a numerical example involving the selection of a logistics specialist, a critical decision in emergency logistics optimization. The research also includes a case study on a logistics specialist, highlighting the practical application of the methods discussed. Furthermore, this study provides a comprehensive sensitivity analysis of the parameters' weights to evaluate the robustness and reliability of the proposed method. The results highlight the effectiveness of the LDF CODAS method in making informed and reliable decisions under conditions of uncertainty, paving the way for future applications in other decision-making scenarios.

Approaches to improve preprocessing for Latent Dirichlet Allocation topic modeling

As a part of natural language processing (NLP), the intent of topic modeling is to identify topics in textual corpora with limited human input. Current topic modeling techniques, like Latent Dirichlet Allocation (LDA), are limited in the pre-processing steps and currently require human judgement, increasing analysis time and opportunities for error. The purpose of this research is to allay some of those limitations by introducing new approaches to improve coherence without adding computational complexity and provide an objective method for determining the number of topics within a corpus. First, we identify a requirement for a more robust stop words list and introduce a new dimensionality-reduction heuristic that exploits the number of words within a document to infer importance to word choice. Second, we develop an eigenvalue technique to determine the number of topics within a corpus. Third, we combine all of these techniques into the Zimm Approach, which produces higher quality results than LDA in determining the number of topics within a corpus. The Zimm Approach, when tested against various subsets of the 20newsgroup dataset, produced the correct number of topics in 7 of 9 subsets vs. 0 of 9 using highest coherence value produced by LDA.

AVN: A Deep Learning Approach for the Analysis of Birdsong.

Deep learning tools for behavior analysis have enabled important new insights and discoveries in neuroscience. Yet, they often compromise interpretability and generalizability for performance, making it difficult to quantitively compare phenotypes across datasets and research groups. We developed a novel deep learning-based behavior analysis pipeline, Avian Vocalization Network (AVN), for the learned vocalizations of the most extensively studied vocal learning model species - the zebra finch. AVN annotates songs with high accuracy across multiple animal colonies without the need for any additional training data and generates a comprehensive set of interpretable features to describe the syntax, timing, and acoustic properties of song. We use this feature set to compare song phenotypes across multiple research groups and experiments, and to predict a bird's stage in song development. Additionally, we have developed a novel method to measure song imitation that requires no additional training data for new comparisons or recording environments, and outperforms existing similarity scoring methods in its sensitivity and agreement with expert human judgements of song similarity. These tools are available through the open-source AVN python package and graphical application, which makes them accessible to researchers without any prior coding experience. Altogether, this behavior analysis toolkit stands to facilitate and accelerate the study of vocal behavior by enabling a standardized mapping of phenotypes and learning outcomes, thus helping scientists better link behavior to the underlying neural processes.

Mapping cognitive biases in multidisciplinary team (MDT) decision-making for cancer care in Scotland: a cognitive ethnography study protocol

IntroductionThe efficiency of multidisciplinary teams (MDTs) in cancer care hinges on facilitating clinicians’ cognitive processes as they navigate complex and uncertain judgements during treatment planning. When systems and workflows are...

AutoTherm: A Dataset and Benchmark for Thermal Comfort Estimation Indoors and in Vehicles

Thermal comfort inside buildings is a well-studied field where human judgment for thermal comfort is collected and may be used for automatic thermal comfort estimation. However, indoor scenarios are rather static in terms of thermal state changes and, thus, cannot be applied to dynamic conditions, e.g., inside a vehicle. In this work, we present our findings of a gap between building and in-vehicle scenarios regarding thermal comfort estimation. We provide evidence by comparing deep neural classifiers for thermal comfort estimation for indoor and in-vehicle conditions. Further, we introduce a temporal dataset for indoor predictions incorporating 31 input signals and self-labeled user ratings by 18 subjects in a self-built climatic chamber. For in-vehicle scenarios, we acquired a second dataset featuring human judgments from 20 subjects in a BMW 3 Series. Our experimental results indicate superior performance for estimations from time series data over single vector input. Leveraging modern machine learning architectures enables us to recognize human thermal comfort states and estimate future states automatically. We provide details on training a recurrent network-based classifier and perform an initial performance benchmark of the proposed dataset. Ultimately, we compare our collected dataset to publicly available thermal comfort datasets.

The principal components of meaning, revisited.

Osgood, Suci, and Tannebaum were the first to attempt to identify the principal components of semantics using dimensional reduction of a high-dimensional model of semantics constructed from human judgments of word relatedness. Modern word-embedding models analyze patterns of words to construct higher dimensional models of semantics that can be similarly subjected to dimensional reduction. Hollis and Westbury characterized the first eight principal components (PCs) of a word-embedding model by correlating them with several well-known lexical measures, such as logged word frequency, age of acquisition, valence, arousal, dominance, and concreteness. The results show some clear differentiation of interpretation between the PCs. Here, we extend this work by analyzing a larger word-embedding matrix using semantic measures initially derived from subjective inspection of the PCs. We then use quantitative analysis to confirm the utility of these subjective measures for predicting PC values and cross-validate them on two word-embedding matrices developed on distinct corpora. Several semantic and word class measures are strongly predictive of early PC values, including first-person and second-person verbs, personal relevance of abstract and concrete words, affect terms, and names of places and people. The predictors of the lowest magnitude PCs generalized well to word-embedding matrices constructed from separate corpora, including matrices constructed using different word-embedding methods. The predictive categories we describe are consistent with Wittgenstein's argument that an autonomous level of social interaction grounds linguistic meaning.

A Language Model-Powered Simulated Patient With Automated Feedback for History Taking: Prospective Study.

Although history taking is fundamental for diagnosing medical conditions, teaching and providing feedback on the skill can be challenging due to resource constraints. Virtual simulated patients and web-based chatbots have thus emerged as educational tools, with recent advancements in artificial intelligence (AI) such as large language models (LLMs) enhancing their realism and potential to provide feedback. In our study, we aimed to evaluate the effectiveness of a Generative Pretrained Transformer (GPT) 4 model to provide structured feedback on medical students' performance in history taking with a simulated patient. We conducted a prospective study involving medical students performing history taking with a GPT-powered chatbot. To that end, we designed a chatbot to simulate patients' responses and provide immediate feedback on the comprehensiveness of the students' history taking. Students' interactions with the chatbot were analyzed, and feedback from the chatbot was compared with feedback from a human rater. We measured interrater reliability and performed a descriptive analysis to assess the quality of feedback. Most of the study's participants were in their third year of medical school. A total of 1894 question-answer pairs from 106 conversations were included in our analysis. GPT-4's role-play and responses were medically plausible in more than 99% of cases. Interrater reliability between GPT-4 and the human rater showed "almost perfect" agreement (Cohen κ=0.832). Less agreement (κ<0.6) detected for 8 out of 45 feedback categories highlighted topics about which the model's assessments were overly specific or diverged from human judgement. The GPT model was effective in providing structured feedback on history-taking dialogs provided by medical students. Although we unraveled some limitations regarding the specificity of feedback for certain feedback categories, the overall high agreement with human raters suggests that LLMs can be a valuable tool for medical education. Our findings, thus, advocate the careful integration of AI-driven feedback mechanisms in medical training and highlight important aspects when LLMs are used in that context.

On the role of the UMLS in supporting diagnosis generation proposed by Large Language Models

Objective:Traditional knowledge-based and machine learning diagnostic decision support systems have benefited from integrating the medical domain knowledge encoded in the Unified Medical Language System (UMLS). The emergence of Large Language Models (LLMs) to supplant traditional systems poses questions of the quality and extent of the medical knowledge in the models’ internal knowledge representations and the need for external knowledge sources. The objective of this study is three-fold: to probe the diagnosis-related medical knowledge of popular LLMs, to examine the benefit of providing the UMLS knowledge to LLMs (grounding the diagnosis predictions), and to evaluate the correlations between human judgments and the UMLS-based metrics for generations by LLMs. Methods:We evaluated diagnoses generated by LLMs from consumer health questions and daily care notes in the electronic health records using the ConsumerQA and Problem Summarization datasets. Probing LLMs for the UMLS knowledge was performed by prompting the LLM to complete the diagnosis-related UMLS knowledge paths. Grounding the predictions was examined in an approach that integrated the UMLS graph paths and clinical notes in prompting the LLMs. The results were compared to prompting without the UMLS paths. The final experiments examined the alignment of different evaluation metrics, UMLS-based and non-UMLS, with human expert evaluation. Results:In probing the UMLS knowledge, GPT-3.5 significantly outperformed Llama2 and a simple baseline yielding an F1 score of 10.9% in completing one-hop UMLS paths for a given concept. Grounding diagnosis predictions with the UMLS paths improved the results for both models on both tasks, with the highest improvement (4%) in SapBERT score. There was a weak correlation between the widely used evaluation metrics (ROUGE and SapBERT) and human judgments. Conclusion:We found that while popular LLMs contain some medical knowledge in their internal representations, augmentation with the UMLS knowledge provides performance gains around diagnosis generation. The UMLS needs to be tailored for the task to improve the LLMs predictions. Finding evaluation metrics that are aligned with human judgments better than the traditional ROUGE and BERT-based scores remains an open research question.

Judgmental adjustment of demand forecasting models using social media data and sentiment analysis within industry 5.0 ecosystems

Industry 5.0 ecosystems focus on a human-centric approach to operations and supply chain management by integrating stakeholders, advanced technologies, and processes. While incorporating social media (SM) information into demand forecasting can significantly improve accuracy, it also brings about several challenges. This paper proposes an approach to leverage Big Data originating from SM networks combined with human judgment to build demand forecasts for new products. The structured methodology is demonstrated to improve forecast accuracy in a real case of a F&B company while providing several insights into the challenges and opportunities of integrating advanced information technology into the demand forecasting process. The main challenges include effectively categorising the impact factors of SM on demand forecasting, translating insights from SM into actionable decisions, and ensuring the accuracy and reliability of the data obtained from SM networks. Future studies should involve collaborative expert input and validating the approach across various companies and industries.

Flight in Transition: Navigating the Skies of Automation and Human Judgement

Integrating automation systems within the aviation sector signifies a critical juncture, presenting significant advancements in operational efficiency and safety protocols. However, this technological evolution mandates a comprehensive reassessment of decision-making processes, particularly as pilots navigate the intricate interaction between human cognition and automated support. This study aims to enhance academic discourse by conducting a thorough analysis of the implications of automation systems on decision-making in aviation, with a specific focus on the tragic incidents involving Lion Air Flight 610 on October 29, 2018, and Ethiopian Airlines Flight 302 on March 10, 2019. The impact of automation systems on decision-making in aviation will be examined in detail to understand their role in these incidents. Employing a rigorous case analysis methodology upon a diverse array of secondary sources, this study seeks to unravel the multifaceted dynamics at play and shed light on the influence of automation on pilot actions and responses. Anticipated outcomes of this study include providing invaluable guidance to aviation stakeholders, encompassing regulatory bodies, aircraft manufacturers, airlines, and pilots, concerning the challenges and opportunities inherent in automation integration. By identifying knowledge gaps and delineating avenues for enhancement, this research endeavours to inform evidence-based strategies and optimal practices for enhancing safety and resilience in contemporary air travel. Ultimately, the significance of this scholarly endeavour lies in its potential to advance scholarly understanding and facilitate informed decision-making processes, thereby contributing to the pursuit of safer and more efficient aviation operations on a global scale.

Examining the merits of feature-specific similarity functions in the news domain using human judgments

Online news article recommendations are typically of the ‘more like this’ type, generated by similarity functions. Across three studies, we examined the representativeness of different similarity functions for news item retrieval, by comparing them to human judgments of similarity. In Study 1 (N=401\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$N=401$$\\end{document}), participants assessed the overall similarity of ten randomly paired news articles on politics and compared their judgments to different feature-specific similarity functions (e.g., based on body text or images). In Study 2, we checked for domain differences in a mixed-methods survey (N=45\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$N=45$$\\end{document}), surfacing evidence that the effectiveness of similarity functions differs across different news categories (‘Recent Events’, ‘Sport’). In Study 3 (N=173\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$N=173$$\\end{document}), we improved the design of Study 1, by controlling for how news articles were matched, differentiating between dissimilar news articles and articles that were matched on a shared topic, named entities, and/or date of publication, across ‘Recent Events’ and ‘Sport’ categories. Across all studies, we found that users mostly used text-based features (e.g., body text, title) for their similarity judgments, while BodyText:TF-IDF was found to be the most representative for their judgments. Moreover, the strength of similarity judgments by humans and similarity scores by feature-specific functions was strongly affected by how news article pairs were matched. We show that humans and similarity functions are better aligned when two news articles are more alike, such as in a news recommendation scenario.

A Novel Out-of-Control Action Plan (OCAP) for Optimizing Efficiency and Quality in the Wafer Probing Process for Semiconductor Manufacturing.

The out-of-control action plan (OCAP) is crucial in the wafer probing process of semiconductor manufacturing as it systematically addresses and corrects deviations, ensuring the high quality and reliability of semiconductor devices. However, the traditional OCAP involves many redundant and complicated processes after failures occur on production lines, which can delay production and escalate costs. To overcome the traditional OCAP's limitations, this paper proposes a novel OCAP aimed at enhancing the wafer probing process in semiconductor manufacturing. The proposed OCAP integrates proactive measures such as preventive maintenance and advanced monitoring technologies, which are tested and verified through a comprehensive experimental setup. Implementing the novel OCAP in a case company's production line reduced machine downtime by over 24 h per week and increased wafer production by about 23 wafers per week. Additionally, probe test yield improved by an average of 1.1%, demonstrating the effectiveness of the proposed method. This paper not only explores the implementation of the novel OCAP but also compares it with the traditional OCAP, highlighting significant improvements in efficiency and production output. The results underscore the potential of advanced OCAP to enhance manufacturing processes by reducing dependency on human judgment, thus lowering the likelihood of errors and improving overall equipment effectiveness (OEE).

Automated Course of Action Evaluation for Military Decision-Making

In future complex and diverse battlefield situations, the existing command system faces the challenge of delayed human judgement of strategy and low objectivity. This paper proposes an artificial intelligence model that takes situation information and course of action simulation results as input and automatically assigns scores to various evaluation elements and a comprehensive score. This tool is expected to assist the commander in making decisions, reduce the time required for making judgments, and promote impartial decision-making.