Human-computer Interaction Paradigm Research Articles

Development of Augmented Reality Platform Using Image Processing with Deep Learning Techniques

The development and structuring of an augmented reality (AR) platform integrating image processing, deep learning, and spatial tracking techniques have yielded promising results. AR technology, which enhances real-world objects with computer-generated information, holds vast potential for education, entertainment, and industry. Image processing enables real-time object detection, recognition, and tracking within the user's environment, while deep learning models enhance pattern recognition accuracy. Spatial tracking techniques ensure seamless integration of virtual content with the physical world, providing immersive AR experiences. Ant colony optimization (ACO) algorithms further enhance the AR platform's functionality by optimizing object placement and interaction. ACO mimics ant behavior to find optimal paths through graphs, effectively optimizing spatial layout and user interaction patterns in dynamic environments. The integration of ACO into the AR platform facilitates efficient pathfinding for augmented objects and enhances the system's responsiveness to user inputs. The successful implementation of the AR platform underscores its potential to revolutionize various industries, including healthcare, education, and marketing. By leveraging AI-driven algorithms and optimization techniques, AR systems can deliver personalized and engaging experiences that blur the boundaries between the digital and physical worlds. These results pave the way for future advancements in AR technology, driving innovation and reshaping human-computer interaction paradigms.

Human-Centered Interaction in Virtual Worlds: A New Era of Generative Artificial Intelligence and Metaverse

The metaverse has emerged as an exciting new paradigm for human-computer interaction (HCI) and virtual collaboration. This paper presents a comprehensive review of the metaverse to address the gap in the existing literature where there is a lack of a survey that reviews the nature of the metaverse and its building blocks from a human-centric perspective. We first synthesize a definition of the metaverse from existing literature and delineate key affordances. We then introduce a detailed framework encompassing the metaverse’s nature, infrastructure technologies, and input/output technologies that facilitate multi-sensory HCI, alongside applications across diverse domains. The components within this framework are explained in depth, offering insights into the metaverse’s nature and the readiness level of current technologies. Based on this comprehensive analysis, we outline major open challenges and propose promising directions demanding further exploration and investigation. By clarifying the vision for the metaverse and characterizing the building blocks required to realize it, this review provides essential insights and serves as an invaluable resource for metaverse developers and researchers working to advance this transformative new medium.

Peripheral focused ultrasound stimulation and its applications: From therapeutics to human-computer interaction.

Peripheral focused ultrasound stimulation (pFUS) has gained increasing attention in the past few decades, because it can be delivered to peripheral nerves, neural endings, or sub-organs. With different stimulation parameters, ultrasound stimulation could induce different modulation effects. Depending on the transmission medium, pFUS can be classified as body-coupled US stimulation, commonly used for therapeutics or neuromodulation, or as an air-coupled contactless US haptic system, which provides sensory inputs and allows distinct human-computer interaction paradigms. Despite growing interest in pFUS, the underlying working mechanisms remain only partially understood, and many applications are still in their infancy. This review focused on existing applications, working mechanisms, the latest progress, and future directions of pFUS. In terms of therapeutics, large-sample randomized clinical trials in humans are needed to translate these state of art techniques into treatments for specific diseases. The airborne US for human-computer interaction is still in its preliminary stage, but further efforts in task-oriented US applications might provide a promising interaction tool soon.

The influence of augmented reality interaction design on Pilot's perceived workload and situation awareness

This work explored the potential for Augmented Reality (AR) rendering information superimposed over the flight deck to increase a pilot's situation awareness (SA). This emerging technology introduced novel human-computer interaction paradigms that would have impact on pilot's cognitive demands. The objective of this research was to evaluate both the pilot's perceived workload and SA while interacting with an AR device using different interactive modes. Participants performed traditional landing checklists as a baseline to compare with the AR gesture-command and voice-command checklists. The research results showed that gesture-commands created additional cognitive and physical demands. Conversely, voice-command checklists could constitute a significant improvement in terms of reducing participants' perceived workload and maximising SA performance. The findings provided evidence that the interactive modes of AR user interface design could influence participant's cognitive information processing and perceived workload in flight operations. However, there were some limitations with AR applications that included latency on response time, narrow field of view, accuracy of voice recognition, calibration within dynamic environment and inexplicable movements of the head position that required further investigation. An AR device can be a great tool for training at the initial stage to increase cost-efficiency in flight operations. Furthermore, the implementation of an AR design may provide part of the potential solution for single pilot operations in the future.

From Strategic Effects to Tactical Affects

When it comes to strategies governing contemporary technological culture, the Human-Computer Interaction (HCI) paradigm is as pervasive and automatic as technology itself. Taking from Michel de Certeau’s distinction between the concepts of strategy and tactics, Ksenia Fedorova analyses a series of transdisciplinary artworks in which computerized operations affecting and transforming human experience are tactically disrupted in order to question technological interfaces mediating HCI. Exposing crossdisciplinary experiments in which affects and deffects are part of the algorithm, Tactics of Interfacing shows precisely in what measure and weight art and technology may contemplate natural and artificial glitches of both human nature and machinic code.

Learning by Doing: SMART CLASSROOMS AND e-LEARNING

This paper is about the much growing technology “SMART CLASSROOMS and e-LEARNING”. The Smart Classroom project explores the challenges and potentials of the Intelligent Environment as a new human-computer interaction paradigm. The Smart Classroom could actively observe, listen and serve the teachers, and teachers can write on a wall-size media-board just by their hands, or use speeches and gestures to conduct the class discussion involving of the distant students. This paper discusses the advantages, disadvantages and main uses of the system.

Towards Domain-independent Complex and Fine-grained Gesture Recognition with RFID

Gesture recognition plays a fundamental role in emerging Human-Computer Interaction (HCI) paradigms. Recent advances in wireless sensing show promise for device-free and pervasive gesture recognition. Among them, RFID has gained much attention given its low-cost, light-weight and pervasiveness, but pioneer studies on RFID sensing still suffer two major problems when it comes to gesture recognition. The first is they are only evaluated on simple whole-body activities, rather than complex and fine-grained hand gestures. The second is they can not effectively work without retraining in new domains, i.e. new users or environments. To tackle these problems, in this paper, we propose RFree-GR, a domain-independent RFID system for complex and fine-grained gesture recognition. First of all, we exploit signals from the multi-tag array to profile the sophisticated spatio-temporal changes of hand gestures. Then, we elaborate a Multimodal Convolutional Neural Network (MCNN) to aggregate information across signals and abstract complex spatio-temporal patterns. Furthermore, we introduce an adversarial model to our deep learning architecture to remove domain-specific information while retaining information relevant to gesture recognition. We extensively evaluate RFree-GR on 16 commonly used American Sign Language (ASL) words. The average accuracy for new users and environments (new setup and new position) are $89.03%$, $90.21%$ and $88.38%$, respectively, significantly outperforming existing RFID based solutions, which demonstrates the superior effectiveness and generalizability of RFree-GR.

Novel virtual reality based training system for fine motor skills: Towards developing a robotic surgery training system.

Training surgeons to use surgical robots are becoming part of surgical training curricula. We propose a novel method of training fine-motor skills such as Microscopic Selection Task (MST) for robot-assisted surgery using virtual reality (VR) with objective quantification of performance. We also introduce vibrotactile feedback (VTFB) to study its impact on training performance. We use a VR-based environment to perform MST with varying degrees of difficulties. Using a well-known human-computer interaction paradigm and incorporating VTFB, we quantify the performance: speed, precision and accuracy. MST with VTFB showed statistically significant improvement in performance metrics leading to faster completion of MST with higher precision and accuracy compared to that without VTFB. The addition of VTFB to VR-based training for robot-assisted surgeries may improve performance outcomes in real robotic surgery. VTFB, along with proposed performance metrics, can be used in training curricula for robot-assisted surgeries.

Study of Ruangguru application based on human computer interaction principles and paradigm

This Ruangguru application study aims to determine the relevance of application functions based on the principles and paradigms of Human and computer Interaction (HCI). In this study, the aspects evaluated included aspects of human, technology, usability, and ergonomics. The interest in studying the Ruangguru application is based on several sources stating that Ruangguru is believed to be the “Best Education Platform”. So from that sentences, it was examined several aspects to determine the extent of the relationship between the Ruangguru application and HCI principle. After the study was conducted, the Ruangguru application was assessed to have fulfilled these aspects. So it’s no wonder the Ruangguru application is used as an online learning centre among Indonesian students.

Understanding the Role of Interactive Machine Learning in Movement Interaction Design

Interaction based on human movement has the potential to become an important new paradigm of human-computer interaction. However, high quality, mainstream movement interaction requires effective tools and techniques to support designers. A promising approach to movement interaction design is Interactive Machine Learning, in which designing is done by physically performing an action. This article brings together many different perspectives on understanding human movement knowledge and movement interaction. This understanding shows that the embodied knowledge involved in movement interaction is very different from the representational knowledge involved in a traditional interface, so a very different approach to design is needed. We apply this knowledge to understand why interactive machine learning is an effective tool for motion interaction designers and to make a number of suggestions for future development of the technique.

HOLMeS: eHealth in the Big Data and Deep Learning Era

Now, data collection and analysis are becoming more and more important in a variety of application domains, as long as novel technologies advance. At the same time, we are experiencing a growing need for human–machine interaction with expert systems, pushing research toward new knowledge representation models and interaction paradigms. In particular, in the last few years, eHealth—which usually indicates all the healthcare practices supported by electronic elaboration and remote communications—calls for the availability of a smart environment and big computational resources able to offer more and more advanced analytics and new human–computer interaction paradigms. The aim of this paper is to introduce the HOLMeS (health online medical suggestions) system: A particular big data platform aiming at supporting several eHealth applications. As its main novelty/functionality, HOLMeS exploits a machine learning algorithm, deployed on a cluster-computing environment, in order to provide medical suggestions via both chat-bot and web-app modules, especially for prevention aims. The chat-bot, opportunely trained by leveraging a deep learning approach, helps to overcome the limitations of a cold interaction between users and software, exhibiting a more human-like behavior. The obtained results demonstrate the effectiveness of the machine learning algorithms, showing an area under ROC (receiver operating characteristic) curve (AUC) of 74.65% when some first-level features are used to assess the occurrence of different chronic diseases within specific prevention pathways. When disease-specific features are added, HOLMeS shows an AUC of 86.78%, achieving a greater effectiveness in supporting clinical decisions.

Analysis and Classification for EEG Patterns of Force Motor Imagery Using Movement Related Cortical Potentials.

Motor imagery-based BCIs are the most natural human-computer interaction paradigms. In recent years, researchers have tried to decode the kinetic information of motor imagery. In this paper, we analyzed and discriminated the EEG patterns of different force levels motor imagery using MRCPs. In the experiment, nine healthy subjects were required to perform the hand force motor imagery tasks (30% MVC and 10% MVC). From the view of MRCPs, the most significant discrimination between the two levels of mental tasks was the manifestation of motor planning. The average classification accuracy for features involving both MRCP and CSP was 78.3%, which was 8.5% higher than the CSP-based features (p¡0.001) and 2% higher than the MRCP-based features. The results demonstrated the feasibility of using MRCPs for hand force motor imagery classification.

Interface paradigm of Bots-based human-computer interaction

Bots, usually known as chatbots, have given rise to a brand-new pattern of user-centered interaction. Owing to the development of multi-modal natural language interaction, Bots no longer rely on the graphical desktop for interaction. Consequently, a new interface paradigm is required to normalize the bots-based human-computer interaction, which is no longer limited to the WIMP (windows, icon, menu, and pointer) paradigm of the desktop paradigm. In this paper, a new ASLI (avatar, scenario, language, and instrument) interface paradigm is proposed on the basis of the extended avatar metaphor. The framework of the Bots-based interaction is provided as well as the corresponding interaction instances, which provide guidance for the interface design of Bots-based interaction. The platforms of Bots and ASLI are also discussed and are believed to be able to derive a Bots-based ecosystem in the near future.

A methodology to develop usability/user experience heuristics

Technology, software systems and human–computer interaction paradigms are evolving. Traditional usability heuristics do not cover all aspects of user–system interactions. Many sets of heuristics have been proposed, with the aim of evaluating specific application domains and their specific usability-related features. In addition, several sets of heuristics are used to evaluate aspects other than usability that are related to the user experience (UX). However, most authors use an informal process to develop usability/UX heuristics; there is no clear protocol for heuristic validation. This can result in sets of usability/UX heuristics that are difficult to understand or use; moreover, the resulting sets of heuristics may not be effective or efficient evaluation tools. This article presents a formal methodology for developing usability/user experience heuristics. The methodology was applied in practice in several case studies; it was also validated through expert opinions.

Effects of mixed-reality on players’ behaviour and immersion in a cultural tourism game: A cognitive processing perspective

Abstract Mixed-reality environments introduce innovative human-computer interaction paradigms assisted by enhanced visual content presentation which require from end-users to perform excessive cognitive tasks related to visual attention, search, processing, and comprehension. In such visually enriched interaction realms, individual differences in perception and visual information processing might affect users’ behaviour and immersion, given that such effects are known to exist in conventional computer environments, like desktop or mobile. In an attempt to shed light on whether, how, and why such effects persist within mixed-reality contexts, we conducted a between-subjects eye-tracking study (N=73) in which users interacted within either a conventional or a mixed-reality technological context, and adopted an accredited cognitive style theory to interpret the derived results. Analysis of results yielded that mixed-reality interaction realms amplified the effects of human cognitive style towards game-specific interaction behaviour and visual behaviour. Findings further support the added value of incorporating human cognitive factors in both design and run-time, aiming to provide adaptive and personalised features to end-users within mixed-reality interaction contexts. Such practical implications are also discussed in this paper.

A survey on human-computer interaction in virtual reality

Human computer interaction is one of the core technologies of virtual reality, which has an important significance on promoting the widely use of virtual reality and improving the user experience. Due to the development of various sensors and hardwares, human computer interaction technology in virtual reality has made remarkable progress. This paper firstly introduces the paradigm of human computer interaction in virtual reality, and then makes a summary about the main research contents and development trends of various virtual reality and augmented reality technologies, including 3D interaction, hand gesture interaction, handheld devices interaction, speech interaction, haptic interaction and multimodal interaction. Finally, the paper proposes some existing problems that need to be further studied.

An HCI paradigm fusing flexible object selection and AOM-based animation

The use of three-dimensional (3D) gesture input devices is important and necessary in 3D systems, but such devices face considerable challenges posed by the high dimensionality of dexterous hand motion. The objective of this study is to achieve real-time interaction in object selection and direct manipulation in 3D application systems by capturing and visualizing the interaction intentions and probing the cognitive behavior models of users. An interactive operation procedure is divided into three stages: object selection, manipulation and reset. Trajectory scene interaction (TSI) is proposed for object selection starting from a fixed position called a forward point (FP). The manipulations exerted on the selected object include grasping and translation. After these manipulations, the gesture is reset to the FP. This work offers four novel contributions. First, flexible object selection and atomic operation model (AOM)-based animations are fused to form a uniform, real-time human-computer interaction (HCI) paradigm. Second, a cognitive behavior model is proposed for recognizing and reacting to hand gestures as captured by a monocular camera. Third, an approach to capturing, expressing, and probing a user's interaction intention is presented. Fourth, a 3D real-time gesture input interface is achieved. The use of the proposed HCI interface, which offers fast speed, satisfactory accuracy and a responsive user experience, is demonstrated in virtual assembly, a game of chess, dialing a cell phone number and menu operation.

TANGIBLE LANDSCAPE: COGNITIVELY GRASPING THE FLOW OF WATER

Complex spatial forms like topography can be challenging to understand, much less intentionally shape, given the heavy cognitive load of visualizing and manipulating 3D form. Spatiotemporal processes like the flow of water over a landscape are even more challenging to understand and intentionally direct as they are dependent upon their context and require the simulation of forces like gravity and momentum. This cognitive work can be offloaded onto computers through 3D geospatial modeling, analysis, and simulation. Interacting with computers, however, can also be challenging, often requiring training and highly abstract thinking. Tangible computing – an emerging paradigm of human-computer interaction in which data is physically manifested so that users can feel it and directly manipulate it – aims to offload this added cognitive work onto the body. We have designed Tangible Landscape, a tangible interface powered by an open source geographic information system (GRASS GIS), so that users can naturally shape topography and interact with simulated processes with their hands in order to make observations, generate and test hypotheses, and make inferences about scientific phenomena in a rapid, iterative process. Conceptually Tangible Landscape couples a malleable physical model with a digital model of a landscape through a continuous cycle of 3D scanning, geospatial modeling, and projection. We ran a flow modeling experiment to test whether tangible interfaces like this can effectively enhance spatial performance by offloading cognitive processes onto computers and our bodies. We used hydrological simulations and statistics to quantitatively assess spatial performance. We found that Tangible Landscape enhanced 3D spatial performance and helped users understand water flow.

TANGIBLE LANDSCAPE: COGNITIVELY GRASPING THE FLOW OF WATER

Abstract. Complex spatial forms like topography can be challenging to understand, much less intentionally shape, given the heavy cognitive load of visualizing and manipulating 3D form. Spatiotemporal processes like the flow of water over a landscape are even more challenging to understand and intentionally direct as they are dependent upon their context and require the simulation of forces like gravity and momentum. This cognitive work can be offloaded onto computers through 3D geospatial modeling, analysis, and simulation. Interacting with computers, however, can also be challenging, often requiring training and highly abstract thinking. Tangible computing – an emerging paradigm of human-computer interaction in which data is physically manifested so that users can feel it and directly manipulate it – aims to offload this added cognitive work onto the body. We have designed Tangible Landscape, a tangible interface powered by an open source geographic information system (GRASS GIS), so that users can naturally shape topography and interact with simulated processes with their hands in order to make observations, generate and test hypotheses, and make inferences about scientific phenomena in a rapid, iterative process. Conceptually Tangible Landscape couples a malleable physical model with a digital model of a landscape through a continuous cycle of 3D scanning, geospatial modeling, and projection. We ran a flow modeling experiment to test whether tangible interfaces like this can effectively enhance spatial performance by offloading cognitive processes onto computers and our bodies. We used hydrological simulations and statistics to quantitatively assess spatial performance. We found that Tangible Landscape enhanced 3D spatial performance and helped users understand water flow.

Virtual reality as assessment tool in psychology.

Virtual environments (VEs), offering a new human-computer interaction paradigm, have attracted much attention, in clinical psychology, especially in the treatment of phobias. However, a possible new application of VR in psychology is as assessment tool: VEs can be considered as an highly sophisticated form of adaptive testing. This chapter describes the context of current psychological assessment and underlines possible advantages of a VR based assessment tool. The chapter also details the characteristics of BIVRS, Body Image Virtual Reality Scale, an assessment tool designed to assess cognitive and affective components of body image. It consists of a non-immersive 3D graphical interface through which the patient is able to choose between 9 figures of different size which vary in size from underweight to overweight. The software was developed in two architectures, the first (A) running on a single user desktop computer equipped with a standard virtual reality development software and the second (B) splitted into a server (B1) accessible via Internet and actually running the same virtual ambient as in (A) and a VRML client (B2) so that anyone can access the application.