Human-system Interaction Research Articles

Metaverse in industrial contexts - a comprehensive review

This paper explores the potential of Metaverse technology in industrial Asset Management (AM). By integrating AI and digital technologies, the Metaverse can enhance Human-System-Interaction (HSI) and optimise AM processes. However, implementing a Metaverse in industrial contexts faces challenges, particularly in visualising physical and virtual assets. This paper conducts a systematic review to address these challenges and identify potential solutions. The findings reveal that while the necessary technologies are available, their widespread adoption in industrial AM is limited. The paper presents a comprehensive overview of research themes related to Metaverse applications in industrial contexts, highlighting the evolving landscape and potential benefits. Ultimately, this research aims to contribute to the advancement of Metaverse technology in industrial AM by providing insights into its development, implementation, and challenges along with an Industrial Metaverse Framework. An example of applying the Metaverse concept in the railway sector has been presented and validated using railway digital assets available within the eMaintenance LAB. The practical implications of this work are expected to result in increased efficiency and effectiveness in the operation and maintenance procedures across various industrial sectors.

Marrying math and mind: towards production planning systems for industry 5.0

How can a sophisticated Production Planning (PP) system help planners? To answer this question, we study the PP literature and focus on human-system interaction. We find that the PP literature mainly focuses on designing sophisticated algorithms. By synthesising forecasting and information system (IS) literature, we provide findings for designing a human-centred PP system. From the forecasting literature, we learn that certain biases and heuristics become relevant when humans work with systems. We categorise this literature into three system elements: information received, delivery system, and decision aid provided, as well as two human factors: cognitive limitations and capabilities. From the IS literature, we learn that different factors impact behaviour over the system acceptance and continued use phases. Behaviour is mainly impacted by the Perceived Ease of Use in the acceptance phase. The Perceived Usefulness mainly determines interventions to the solutions in the continued use phase. We combine the two phases distinguished in the IS literature with the three system elements distinguished in the forecasting literature and thus propose a two-phase × three-element matrix for designing human-centred systems, obtain findings that serve as guidelines for designing human-centred PP systems, and propose an agenda for future research in PP.

From human-system interaction to human-system co-action and back: ethical assessment of generative AI and mutual theory of mind

AbstractHuman-machine ethics has emerged as a rapidly growing research field in recent years. However, it seems that Generative Artificial Intelligence (AI) leads to a paradigm shift from human-machine interaction to co-action. The ethical assessment of such relationships is still in the making and needs further scrutiny. First, studies about the influence of technology in human-system interactions and manipulation are reviewed. Second, the “mutual theory of mind” approach is critically examined to identify its shortcomings. Third, creating user models is reconstruced to demonstrate the strategies of systems. Finally, use cases are discussed and assessed to outline ethical implications.

Review of the opportunities and challenges to accelerate mass‐scale application of smart grids with large‐language models

AbstractSmart grids represent a paradigm shift in the electricity industry, moving from traditional one‐way systems to more dynamic, interconnected networks. These grids are characterised by their intelligent automation, robust structure, and enhanced interaction with customers, backed by comprehensive monitoring and data analytics. The key of this transformation is the integration of data‐driven methods into smart grids. Compared to previous big data solutions, large language models (LLMs), with their advanced generalisation abilities and multi‐modal competencies, are crucial in effectively managing and integrating diverse data sources. They address challenges such as data inconsistency, inadequate quality, and heterogeneity, thereby enhancing the operational efficiency and reliability of smart grids. Furthermore, at the system level, LLMs improve human–system interactions, making smart grids more user‐friendly and intuitive. Last but not the least, the structure of LLMs performs inherent advantages in bolstering system security and privacy, alongside in resolving issues related to system compatibility and integration. The paper reviews the data‐empowered smart grids and for the first time finds and proposes opportunities and future directions for adopting LLMs to accelerate the mass‐scale application of Smart Grids.

Exploring the Impact of AI Enhancement on the Sports App Community: Analyzing Human-Computer Interaction and Social Factors Using a Hybrid SEM-ANN Approach

The iteration of AI technology has enhanced the industry competitiveness of sports apps and brought a new emotional experience to users. However, the community function of AI-enabled sports apps has not yet received sufficient attention. This study adopts the technology acceptance model framework through the human-computer interaction and social psychology perspectives, incorporating perceived interactivity (human-human interaction, human-information interaction, human-system interaction), user experience (functional experience, content experience, and emotional experience), attachment theory (emotional attachment, emotional loyalty), social identity, social trust, and satisfaction variables to construct a theoretical model. It aims to solve the problem of how AI can enhance users’ emotional interaction, community stickiness, and ultimately social interaction satisfaction in sports apps. Through literature review, theoretical analysis, and case study, the Chinese “KEEP” APP was identified as the research object, and the questionnaire was constructed with an improved test scale. The study used a cross-sectional sampling method to collect data from Chinese online platforms for three months (N = 520) and tested the hypotheses in the theoretical model through structural equation modeling and artificial neural networks. The results of the study show that (1) emotional attachment, emotional loyalty, social identity, and social trust positively and significantly affect user satisfaction; (2) emotional attachment and social identity positively and significantly affect human-human interaction; (3) perceived usefulness, social identity, and social trust significantly construct an influence loop on the internal path. Compared with existing studies, this study proposes a multidimensional innovative framework that reveals the deeper influence mechanism of AI technology in sports APP communities and makes up for the shortcomings of existing studies in terms of users’ emotional interactions and community stickiness. Through this study, sports APP developers and operators, marketers, AI technology developers, as well as sports enthusiasts, and general users can obtain specific practical suggestions to further optimize the design and functionality of sports apps, and enhance the overall user experience and satisfaction.

Flexible wide-range, sensitive three-axis pressure sensor array for robotic grasping feedback

Flexible pressure sensors capable of detecting normal and tangential forces through physical contact have garnered considerable interest in the realm of human-interactive systems. However, simultaneous detection of multi-directional forces is still a challenge for current research. Herein, a capacitive flexible pressure sensor based on a sandwich structure for three-dimensional force detection is proposed. The fabrication process of the sensor array is straightforward, capable of effectively distinguishing between normal and tangential forces. Polyimide (PI) serves as the flexible substrate for depositing the metal electrode pattern, while Polydimethylsiloxane (PDMS) acts as the intermediate dielectric layer material and the three-dimensional force conduction block. Through a comparative study of the thickness of the hollow dielectric layer, a pressure sensor with superior performance was prepared, featuring high sensitivity across a wide working range. Test results demonstrate its capability to detect normal forces ranging from 0 to 46 N (0–520 kPa) with a sensitivity of 0.442 N−1 (0.031 kPa−1) and tangential forces from 0 to 10 N with a sensitivity of 0.08 N−1 (X-axis) and 0.07 N−1 (Y-axis). The designed acquisition system can simultaneously gather data from 6 sensor arrays, totaling 240 channels, with a response time of 11 ms. This sensor array, characterized by flexibility, versatility, and a wide range, is suitable for applications in robot tactile perception.

Digital Human Intelligent Interaction System Based on Multimodal Pre-training Mode

ABSTRACT As the forefront of human–computer intelligent interaction, digital humans have increasingly diverse application scenarios, but also face many challenges. In order to enhance the intelligence level and interaction capability of digital human, the study first optimizes the traditional UniLM model, then introduces the mechanism of multi-head attention, and mitigates the exposure bias by using adversarial training and random replacement of decoder to design an improved UniLM model, and finally applies the improved model to the digital human management system. The results show that the precision rate of the improved UniLM model is improved by 7.68%, 6.4%, and 4.96%, the recall rate is improved by 11.94%, 9.69%, and 8.83%, and the F1-score is improved by 8.34%, 6.41%, and 7.68% compared with the other three models, which proves that it has a better precision rate, robustness, and generalization ability. The perplexity of the improved UniLM model is 135, 95, 76, 71, and 55 under five text lengths, which is significantly lower than the other models, proving that its text generation ability is better. The above results demonstrate the performance of the research-designed digital human system based on the Improved UniLM model, which provides a direction for the further development of digital human technology.

Smart All-Hazard Responses Framework (SARF): Human Factors and Ergonomics Approach

The integration of Human Factors and Ergonomics (HFE) into emergency responses is crucial due to frequent human errors and inadequate human-system interactions which impede effective emergency and disaster management. Therefore, this research aims to develop and validate the Smart All-hazards Response Framework (SARF) that systematically incorporates HFE principles across all aspects of disaster management, from planning through post-response analysis. The framework focuses on reducing human errors, enhancing human-system interactions, and boosting system resilience. The research methodology includes a detailed literature review, the development of the HFE-integrated framework, and rigorous testing through experimental studies and simulations across various scenarios. Key questions address the integration of HFE to optimize system performance and the impact of ergonomic interventions on responder safety in high-risk environments. By advancing a holistic, human-centered approach to emergency and disaster management, this study aims to significantly enhance the effectiveness, safety, and adaptability of responses to all kinds of hazards.

1B4-2 Issues on human-system interaction with visual displays and Image Safety, and their International Standardizations

1B4-2 Issues on human-system interaction with visual displays and Image Safety, and their International Standardizations

Exploring Trade-offs in Explainable AI

Machine Learning (ML) models are increasingly used in systems that involve physical human interaction or decision-making systems that impact human health and safety. Ensuring that these systems are safe and reliable is an important topic of current AI research. For many ML models it is unclear how a prediction (output) is arrived at from the provided features (input). Critical systems cannot blindly trust the predictions of such "black box" models, but instead need additional reassurance via insight into the model's reasoning. A range of methods exist within the field of Explainable AI (XAI) to make the reasoning of black box ML models more understandable and transparent. The explanations provided by XAI methods may be evaluated in a number of (competing) ways. In this paper, we investigate the trade-off between selected metrics for an XAI method called UnRAvEL, which is similar to the popular LIME approach. Our results show that by weighting the terms within the acquisition function used in UnRAvEL, different trade-offs can be achieved.

Examining the Use of Intelligent Conversational Voice-Assistants for Improved Mobility Behavior of Older Adults in Smart Cities

Intelligent Conversational Voice-assistants (ICV) can interpret the speech of human and respond back using synthesized voices and are increasingly employed for interacting with different applications. ICV offers promising tool to address the increasing mobility accessibility challenge faced by older people. Yet, irrespective of the increased ageing population, studies that employed ICV to improve the mobility, walkability, and wayfinding of older people are scarce. This is because the development of ICV in the mobility domain presents several designs, technical, and linguistic challenges. A systematic literature review was adopted grounded on secondary data from the literature and descriptive analysis was employed. Grounded on user-centred design perspective based on the “ability-based design framework” and “ISO 9241-110 framework for ergonomics of human-system interaction.” Therefore, this study describes the requirement specifications needed to design an ICV suitable for supporting the mobility behavior of older people. Findings from this paper identifies the perceived factors that influences the use of ICV by older people. Additionally, findings from this study discusses how ICV that simulates human-like behavior can provide personalized mobility, walkability, and wayfinding guide when older people walk in cities. This study provides recommendations for the implementation of a user-centred ICV for safe, independent, accessible mobility for older people in urban environment.

Design and Optimization of Human-Computer Interaction System for Education Management Based on Artificial Intelligence

The continuous improvement and refinement of artificial intelligence (AI) technology has facilitated the broader application of human-computer interaction in the field of education management. The construction of an educational management human-computer interaction system based on AI technology can optimize and improve key parameters of educational management human-computer interaction scenarios, thereby creating a more comprehensive mobile learning (m-learning) application system. This paper is based on AI technology, analyzing gesture semantics and speech semantics, and combining fusion algorithms to construct an education management human interaction system. The performance changes of the system were compared with real experimental operations and the NOBOOK platform analysis. The results show that the education management human-computer interaction system constructed in this article can enhance the m-learning experience of participants. It ensures high recognition accuracy, leading to higher scores in all dimensions of indicator evaluation. Therefore, as one of the crucial forms of m-learning, the human-computer interaction system for education management based on AI can establish a foundation for the further enhancement and development of education management.

Portable Head-Mounted System for Mobile Forearm Tracking.

Computer vision (CV)-based systems using cameras and recognition algorithms offer touchless, cost-effective, precise, and versatile hand tracking. These systems allow unrestricted, fluid, and natural movements without the constraints of wearable devices, gaining popularity in human-system interaction, virtual reality, and medical procedures. However, traditional CV-based systems, relying on stationary cameras, are not compatible with mobile applications and demand substantial computing power. To address these limitations, we propose a portable hand-tracking system utilizing the Leap Motion Controller 2 (LMC) mounted on the head and controlled by a single-board computer (SBC) powered by a compact power bank. The proposed system enhances portability, enabling users to interact freely with their surroundings. We present the system's design and conduct experimental tests to evaluate its robustness under variable lighting conditions, power consumption, CPU usage, temperature, and frame rate. This portable hand-tracking solution, which has minimal weight and runs independently of external power, proves suitable for mobile applications in daily life.

Analysis of human errors in human-autonomy collaboration in autonomous ships operations through shore control experimental data

Human-autonomy collaboration plays a pivotal role in the development of Maritime autonomous surface ships (MASS), as Shore control center (SCC) operators may engage in the control loop by directly operating the MASS, or, in the supervisory loop, monitoring the MASS and taking over control when needed. Thus, effective human performance during takeover control and operation is crucial for the safety of MASS operations. However, since the MASS is still in the early phase of development, the mechanism of human errors is unknown, and the data on human-autonomy collaborative operation is scarce. Human reliability analysis (HRA) aims to assess human errors qualitatively and quantitatively, and is widely used in various complex systems to help safety analysis. This study is dedicated to incorporating advanced HRA methods elements to identify and quantify human errors during taking over control and operation of a MASS in collision avoidance scenarios. It presents virtual experimental results, combined with theoretical human error identification and assessment methods. At first, we apply the Human-System Interaction in Autonomy (H-SIA) method to identify potential human errors; secondly, we identify relevant Performance Shaping Factors (PSFs) including Experience, Boredom, Task complexity, Available time and Pre-warning, and performance measures of the human errors, and implement them in the virtual experiment based on a full-scale autonomous ferry research vessel called milliAmpere2. Finally, we build a Bayesian Network (BN) to present causal and probabilistic relationships between PSFs and human errors through experimental data. The results show that available time has the highest impact on takeover performance of operators, followed by task complexity and pre-warning. Boredom does not present a significant sole impact unless combined with available time. Experience does not show a significant impact on human performance. In addition to the relevance of the human errors analysis to the safe development and operational design of MASS, the developed method benefits other human-autonomy collaborative systems. The developed BN model shows adaptability to assess human error probabilities, and the practical significance of integrating experimental data into the existing HRA methodologies for complex systems.

Interactive Learning System for Learning Calculus.

IT tools has brought a new perspective to collaborative learning where students do not just sit in a chair and swallow lecture content but instead participate in creating and sharing knowledge. However, calculus learning augmented reality application has limitation in promoting a human collaboration in learning. This research develops an interactive application for learning calculus that promotes human-system interaction via augmented reality (AR) and human-human interaction through chat functions. The study examines the effect of both interactivities on learning experience and how that learning experience affects the performance of learning. The research adopted a quasi-experimental study design and pre-post test data analysis to evaluate the effect of interactivities on learning experience and consequently the effect of learning experience on learning performance. The subjects were exposed to the developed application for learning the calculus chapter "Solid of Revolution" in a controlled environment. The study validated its research framework through partial least squares path modelling and tested three hypotheses via pre-and post-test evaluation. The results found that both interactivities affect learning experience positively; human-human interactivity has a higher impact than the human-system interactivity. It was also found that learning performance as part of the learning experience increased from pre-test to post-test.

Safety protocols in electro-mechanical installations: Understanding the key methodologies for ensuring human and system safety

This study systematically examines the safety protocols in electro-mechanical systems, focusing on methodologies, trends, and innovations that ensure human and system safety. The primary objective is to bridge the gap between current practices and the evolving demands of these systems, considering technological advancements and regulatory compliance. A systematic literature review methodology was employed, utilizing databases like IEEE Xplore, ScienceDirect, and Google Scholar. The search strategy involved keywords related to safety protocols and electro-mechanical systems, adhering to strict inclusion and exclusion criteria to ensure a focused analysis. Key findings reveal the growing importance of integrating advanced sensing technologies, artificial intelligence, and human factors into safety management. The study highlights the shift from reactive to proactive safety measures, emphasizing the need for continuous monitoring and predictive maintenance. Collaborative approaches involving various stakeholders, including industry and academia, are identified as crucial for enhancing safety. The future landscape of safety in electro-mechanical systems is expected to be significantly influenced by emerging technologies, leading to more sophisticated safety protocols. The study recommends continuous updating of safety protocols in line with technological advancements and changing regulatory landscapes. It also suggests interdisciplinary research and industry-academia collaboration to address complex safety challenges. Finally, the evolution of safety protocols in electro-mechanical systems is marked by technological advancements and a deeper understanding of human-system interaction. The study underscores the necessity for continuous improvement and adaptation of safety protocols, ensuring their effectiveness in safeguarding human lives and system sustainability.

Mental Model Matrix: Implications for System Design and Training

Mental models are characterized as a person’s mental representation of how something works, guiding how they process information, anticipate future events, and interact with devices and tools in complex sociotechnical systems. This paper introduces the Mental Model Matrix (MMM), a novel framework for conceptualizing mental models in the context of human-system interactions. The MMM framework partitions aspects of an individual’s knowledge about a target system into two primary dimensions (system and user), each containing beliefs regarding capabilities and limitations. The system-based dimension includes knowledge for the system’s parts, connections, and causal relationships (capabilities), as well as knowledge for how the system can fail (limitations). The user-based dimension incorporates tacit knowledge for working with the system (capabilities) along with an appreciation for one’s own confusions and misunderstandings (limitations). The MMM provides a conceptual framework and guidance for end-users, trainers, and system designers seeking to elicit and codify different dimensions of an individual’s mental model of the systems they work with. The MMM can be used to reveal knowledge gaps and misalignments between different stakeholders in an organization, which can facilitate the development of human-centered technologies and systems, and the creation of effective training programs.

Mechanochromic and ionic conductive cholesteric liquid crystal elastomers for biomechanical monitoring and human-machine interaction.

Cholesteric liquid crystal elastomers (CLCEs) that combine rubbery elasticity with structural colour from self-assembled helical nanostructures are of paramount importance for diverse applications such as biomimetic skins, adaptive optics and soft robotics. Despite great advances, it is challenging to integrate electrical sensing and colour-changing characteristics in a single CLCE system. Here, we report the design and synthesis of an ionic conductive cholesteric liquid crystal elastomer (iCLCE) through in situ Michael addition and free-radical photopolymerization of CLCE precursors on silane-functionalized polymer ionic liquid networks, in which robust covalent chemical bonding was formed at the interface. Thanks to superior mechanochromism and ionic conductivity, the resulting iCLCEs exhibit dynamic colour-changing and electrical sensing functions in a wide range upon mechanical stretching, and can be used for biomechanical monitoring during joint bending. Importantly, a capacitive elastomeric sensor can be constructed through facilely stacking iCLCEs, where the optical and electrical dual-signal reporting performance allows intuitive visual localization of pressure intensity and distribution. Moreover, proof-of-concept application of the iCLCEs has been demonstrated with human-interactive systems. The research disclosed herein can provide new insights into the development of bioinspired somatosensory materials for emerging applications in diverse fields such as human-machine interaction, prostheses and intelligent robots.

Time series synchronization in cross-recurrence networks: uncovering a homomorphic law across diverse complex systems

Exploring the synchronicity between time series, especially the similar patterns during extreme events, has been a focal point of research in academia. This is due to the fact that such special dependence occurring between pairs of time series often plays a crucial role in triggering emergent behaviors in the underlying systems and is closely related to systemic risks. In this paper, we investigate the relationship between the synchronicity of time series and the corresponding topological properties of the cross-recurrence network (CRN). We discover a positive linear relationship between the probability of pairwise time series event synchronicity and the corresponding CRN’s clustering coefficient. We first provide theoretical proof, then demonstrate this relationship through simulation experiments by coupled map lattices. Finally, we empirically analyze three instances from financial systems, Earth’s ecological systems, and human interactive behavioral systems to validate that this regularity is a homomorphic law in different complex systems. The discovered regularity holds significant potential for applications in monitoring financial system risks, extreme weather events, and more.

Bridging artificial intelligence and cognitive ergonomics: a call for synergy

There is considerable potential between the field of artificial intelligence (AI) and cognitive ergonomics. The purpose of this thesis is to support increasing attention to the synergy between artificial intelligence and cognitive ergonomics, as well as the urgent need to explore it in research and practice.Artificial intelligence (AI) has become an integral part of the modern workplace, transforming industries, increasing productivity, and streamlining processes. However, as AI technologies continue to evolve, it is essential to consider the human side of this digital transformation. Ergonomics, the science of designing workspaces and tools that fit the people who use them, is a critical element in ensuring that the integration of artificial intelligence into the workplace increases employee well-being, efficiency, and overall success.
 With its rapid advancements, artificial intelligence is revolutionizing various fields and industries, from healthcare to self-driving vehicles and financial services. While the potential benefits are obvious, the ethical, cognitive, and ergonomic implications are just as important. Cognitive ergonomics is a sub-branch of ergonomics that deals with the design and evaluation of systems and technologies that support human cognitive processes such as perception, attention, memory, and decision-making. The field of cognitive ergonomics, which primarily focuses on Optimizing human-system interaction is the key to ensuring that artificial intelligence systems are designed and implemented in a way that aligns with human capacities, abilities, and cognitive limitations . Current AI implementations typically adopt a technology-driven focus, and employees are expected to adapt to the technology. In this technology-driven focus, the performance, performance, and accuracy of AI are optimized, but these aspects are considered separately. This point of view raises various critical considerations that are often neglected in the design and implementation of advanced technologies and sometimes have disastrous consequences. From the point of view of ergonomics, the design of artificial intelligence should be transferred from a technology-oriented focus to a systemic and human perspective. By applying a focus on personnel, artificial intelligence should be meaningfully and safely designed and integrated into work processes with the aim of optimizing overall system performance and people's well-being.