Multimodal Interaction Technologies Research Articles

Editorial: Multimodal interaction technologies for mental well-being

Editorial: Multimodal interaction technologies for mental well-being

A Table Tennis Motion Correction System Based on Human Motion Feature Recognition

Computer-aided sports systems are an important area of current research. To be specific, it can combine computer vision, computer graphics, and motion capture technologies with the characteristics of sports, and design and develop assisted systems according to user needs. As a result, it is of great practical importance and application to improve the entertainment and science of sports learning, to stimulate public participation in sports, and to learn sports skills. Table tennis has a very great popular base in China and has a wide range of audiences. However, for beginners or amateurs of table tennis, it is difficult to learn table tennis skills by relying on books, videos, etc., due to the lack of professional technical guidance, which makes it relatively difficult for them to continuously improve their skills. Sometimes they tend to form wrong technical habits, which are not easy to detect and correct in time, and can lead to sports injuries over time. Therefore, the lack of professional instruction has hindered the development of table tennis in public fitness to a certain extent. On the other hand, in a table tennis course, the learning of table tennis techniques is mostly dependent on the teacher’s explanation. Students are bored with the repetitive and boring teaching because they need to repeat the exercises several times. The formation of correct table tennis movements requires repeated instruction from the instructor to address problems with the student’s movements. However, due to limited teaching resources, it is difficult for teachers to accommodate different students during the teaching process. In addition, with a limited team of instructors, the level of proficiency varies, making it difficult for them to make an objective assessment. As a result, with the rapid development and application of modern science and technology in the field of sports, there is an urgent need to explore intelligent sports technology instructional support systems to improve the shortcomings of the traditional teaching model. Motion recognition technology, as an essential branch of multimodal interaction technology, has been developing rapidly in recent years. This research focus on designing and implementing a system for recognizing human motion based on skeletal point pose information by combining inertial sensors. This system can collect data from sensors located at the main skeletal points of the human body and transmits them to the host computer through multi-Bluetooth pairing transmission. After that, support vector machines are applied to classify human movements and to recognize general human movements. This system has significant advantages for human movement recognition and classification due to the distinctive technical characteristics of table tennis. The recognition and classification of both players in the video can play an important role in the technical analysis and tactical arrangement of the players. As a result, the table tennis motion correction system based on human motion feature recognition has crucial research significance and application value.

Design of Sports Training Simulation System for Children Based on Improved Deep Neural Network.

With the development of AI technology, human-computer interaction technology is no longer the traditional mouse and keyboard interaction. AI and VR have been widely used in early childhood education. In the process of the slow development and application of voice interaction, visual interaction, action interaction, and other technologies, multimodal interaction technology system has become a research hotspot. In this paper, dynamic image capture and recognition technology is integrated into early childhood physical education for intelligent interaction. According to the basic movement process and final node matching in children's sports training to judge children's physical behavior ability, attention is paid to identify the accuracy and safety of movement. The input images and questions are from the abstract clipart dataset of dynamic image recognition and the self-made 3D dataset of Web3D dynamic motion scene with the same style, which is similar to the action content in the actual preschool training teaching. Therefore, according to the idea of process capture and target recognition, on the basis of the original conditions of the recognition model, a new recognition model is developed through Zheng's target detector. The modified model is characterized by higher accuracy. Weapons need to combine process recognition and result recognition. The experimental results show that the improved model has the obvious advantages of high precision and fast speed, which provides a new research idea for the development of children's physical training simulation.

Technologies for Multimodal Interaction in Extended Reality—A Scoping Review

When designing extended reality (XR) applications, it is important to consider multimodal interaction techniques, which employ several human senses simultaneously. Multimodal interaction can transform how people communicate remotely, practice for tasks, entertain themselves, process information visualizations, and make decisions based on the provided information. This scoping review summarized recent advances in multimodal interaction technologies for head-mounted display-based (HMD) XR systems. Our purpose was to provide a succinct, yet clear, insightful, and structured overview of emerging, underused multimodal technologies beyond standard video and audio for XR interaction, and to find research gaps. The review aimed to help XR practitioners to apply multimodal interaction techniques and interaction researchers to direct future efforts towards relevant issues on multimodal XR. We conclude with our perspective on promising research avenues for multimodal interaction technologies.

The Enlightenment of the Theory Model of Audio-Visual Integration to Human-Computer Interaction Technology

The visual and auditory systems are the most important information processing systems in the brain. We usually sense external stimuli in the way of audio-visual integration. In this process, “visual” and “auditory” nervous form two functional systems which are relatively independent and interactive. It is of great significance for the fusion of multi-sensor data and human-computer interaction technology to understand the theoretical principles of integrating visual and auditory cues and the neural mechanism of audio-visual integration. Here, we try to explore new methods of audio-visual integration and develop new potential of human-computer interface. Previous studies have shown that the principles of integrating visual and auditory cues mainly include spatio-temporal rule, inverse-effectiveness rule and reliability rule. On this basis, researchers have gradually developed a theoretical model of audio-visual integration—causal inference model. By clarifying the mechanism of audio-visual integration, it provides a theoretical basis for the multimodal interaction technology of human-computer interaction, so as to improve the naturalness and efficiency of human-computer interaction and overcome the shortcomings of traditional human-computer interaction.

Human augmentation: Past, present and future

Abstract Human augmentation is a field of research that aims to enhance human abilities through medicine or technology. This has historically been achieved by consuming chemical substances that improve a selected ability or by installing implants which require medical operations. Both of these methods of augmentation can be invasive. Augmented abilities have also been achieved with external tools, such as eyeglasses, binoculars, microscopes or highly sensitive microphones. Lately, augmented reality and multimodal interaction technologies have enabled non-invasive ways to augment human. In this article, we first discuss the field and related terms. We provide relevant definitions based on the present understanding of the field. This is followed by a summary of existing work in augmented senses, action, and cognition. Our contribution to the future includes a model for wearable augmentation. In addition, we present a call for research to realize this vision. Then, we discuss future human abilities. Wearable technologies may act as mediators for human augmentation, in the same manner as eyeglasses once revolutionized human vision. Non-invasive and easy-to-use wearable extensions will enable lengthening the active life for aging citizens or supporting the full inclusion of people with special needs in society, but there are also potential problems. Therefore, we conclude by discussing ethical and societal issues: privacy, social manipulation, autonomy and side effects, accessibility, safety and balance, and unpredictable future.

Exploring Emergent Features of Student Interaction within an Embodied Science Learning Simulation

Theories of embodied cognition argue that human processes of thinking and reasoning are deeply connected with the actions and perceptions of the body. Recent research suggests that these theories can be successfully applied to the design of learning environments, and new technologies enable multimodal platforms that respond to students’ natural physical activity such as their gestures. This study examines how students engaged with an embodied mixed-reality science learning simulation using advanced gesture recognition techniques to support full-body interaction. The simulation environment acts as a communication platform for students to articulate their understanding of non-linear growth within different science contexts. In particular, this study investigates the different multimodal interaction metrics that were generated as students attempted to make sense of cross-cutting science concepts through using a personalized gesture scheme. Starting with video recordings of students’ full-body gestures, we examined the relationship between these embodied expressions and their subsequent success reasoning about non-linear growth. We report the patterns that we identified, and explicate our findings by detailing a few insightful cases of student interactions. Implications for the design of multimodal interaction technologies and the metrics that were used to investigate different types of students’ interactions while learning are discussed.

A Review of Haptic Perception: Focused on Sensation and Application

Objective: The aim of this study is to investigate haptic perception related researches into three perspectives: cutaneous & proprioceptive sensations, active & passive touch, and cognition & emotion, then to identify issues for implementing haptic interactions. Background: Although haptic technologies had improved and become practical, more research on the method of application is still needed to actualize the multimodal interaction technology. Systematical approached to explore haptic perception is required to understand emotional experience and social message, as well as tactile feedback. Method: Content analysis were conducted to analyze trend in haptic related research. Changes in issues and topics were investigated using sensational dimensions and the different contents delivered via tactile perception. Result: The found research opportunities were haptic perception in various body segments and emotion related proprioceptive sensation. Conclusion: Once the mechanism of how users perceives haptic stimuli would help to develop effective haptic interactrion and this study provide insights of what to focus for the future of haptic interaction. Application: This research is expected to provide presence, and emotional response applied by haptic perception to fields such as human-robot, human-device, and telecommunication interaction.

ALL-N-022 マルチモーダルインタラクションにより対人追従するロボット(お供ロボット)の研究開発(NEDO次世代ロボット実用化プロジェクト(プロトタイプ開発支援事業),生活を支援するロボメカ技術のメガインテグレーション)

ALL-N-022 マルチモーダルインタラクションにより対人追従するロボット(お供ロボット)の研究開発(NEDO次世代ロボット実用化プロジェクト(プロトタイプ開発支援事業),生活を支援するロボメカ技術のメガインテグレーション)

Les découvreurs de l’hybridation

Human augmentation is a field of research that aims to enhance human abilities through medicine or technology. This has historically been achieved by consuming chemical substances that improve a selected ability or by installing implants which require medical operations. Both of these methods of augmentation can be invasive. Augmented abilities have also been achieved with external tools, such as eyeglasses, binoculars, microscopes or highly sensitive microphones. Lately, augmented reality and multimodal interaction technologies have enabled non-invasive ways to augment human.In this article, we first discuss the field and related terms. We provide relevant definitions based on the present understanding of the field. This is followed by a summary of existing work in augmented senses, action, and cognition. Our contribution to the future includes a model for wearable augmentation. In addition, we present a call for research to realize this vision. Then, we discuss future human abilities. Wearable technologies may act as mediators for human augmentation, in the same manner as eyeglasses once revolutionized human vision. Non-invasive and easy-to-use wearable extensions will enable lengthening the active life for aging citizens or supporting the full inclusion of people with special needs in society, but there are also potential problems. Therefore, we conclude by discussing ethical and societal issues: privacy, social manipulation, autonomy and side effects, accessibility, safety and balance, and unpredictable future.