Multimodal Interactive Applications Research Articles

A quantitative study on the effects of an interactive multimodal application to promote students' learning motivation and comprehension in studying Tang poetry.

Studying China's Tang poetry is a crucially integrated part of the language curriculum in primary schools because it is an important part of its cultural heritage and classical literature. However, due to the fact that Tang poetry is written in classical Chinese, which is quite different from modern Chinese Mandarin, and the complex categories of this poetry style, learning Tang poetry can be a challenging experience for many students. To address this problem, this study developed an interactive multimodal application based on the cognitive-affective theory of learning with media to learn Tang poetry in an interactive way. In order to assess the effectiveness of this method, a pretest-posttest control group experiment was conducted. The experiment included eighty third-grade students randomly and equally divided into experimental and control groups from an elementary school in Xinzheng, Henan Province, to test (1) whether the interactive multimodal application improves students' reading comprehension of Tang poetry and (2) whether the application enhances students' intrinsic and/or extrinsic motivation in learning Tang poetry. A multimodal interactive application was used by the experimental group to learn Tang poetry, while the control group used a traditional classroom method. According to the study's findings, it was found that students' intrinsic motivation and comprehension of Tang poetry improved through the use of the interactive multimodal application mode.

Multimodal interaction and IoT applications

Multimodal interaction enables multiple modes for users to interact with the system. Whereas the multimodal interaction with IoT applications depends on multimedia systems’ input/output which may limit their scalability and expressiveness. Furthermore, multimodal interaction with IoT applications could also be limited by input/output integration, security, privacy, and portability issues. This special issue focuses on recent state-of-the-art advances in multimodal interaction and IoT applications that could address these challenges. We aim at bringing together the latest industrial and academic progress, research, and development efforts within the rapidly maturing multimodal interaction and IoT applications. This article summarizes the research contribution of the accepted papers (27.94% acceptance rate) along with possible future directions emanating from these papers.

A cloud-based middleware for multi-modal interaction services and applications

Smart devices, such as smart phones, voice assistants and social robots, provide users with a range of input modalities, e.g., speech, touch, gestures, and vision. In recent years, advancements in processing of these input channels enable more natural interaction (e.g., automated speech, face, and gesture recognition, dialog generation, emotion expression etc.) experiences for users. However, there are several important challenges that need to be addressed to create these user experiences. One challenge is that most smart devices do not have sufficient computing resources to execute the Artificial Intelligence (AI) techniques locally. Another challenge is that users expect responses in near real-time when they interact with these devices. Moreover, users also want to be able to seamlessly switch between devices and services any time and from anywhere and expect personalized and privacy-aware services. To address these challenges, we design and develop a cloud-based middleware (CMI) which helps to develop multi-modal interaction applications and easily integrate applications to AI services. In this middleware, services developed by different producers with different protocols and smart devices with different capabilities and protocols can be integrated easily. In CMI, applications stream data from devices to cloud services for processing and consume the results. It supports data streaming from multiple devices to multiple services (and vice versa). CMI provides an integration framework for decoupling the services and devices and enabling application developers to concentrate on “interaction” instead of AI techniques. We provide simple examples to illustrate the conceptual ideas incorporated in CMI.

Data fusion methods in multimodal human computer dialog

In multimodal human computer dialog, non-verbal channels, such as facial expression, posture, gesture, etc, combined with spoken information, are also important in the procedure of dialogue. Nowadays, in spite of high performance of users’ single channel behavior computing, it is still great challenge to understand users’ intention accurately from their multimodal behaviors. One reason for this challenge is that we still need to improve multimodal information fusion in theories, methodologies and practical systems. This paper presents a review of data fusion methods in multimodal human computer dialog. We first introduce the cognitive assumption of single channel processing, and then discuss its implementation methods in human computer dialog; for the task of multi-modal information fusion, serval computing models are presented after we introduce the principle description of multiple data fusion. Finally, some practical examples of multimodal information fusion methods are introduced and the possible and important breakthroughs of the data fusion methods in future multimodal human-computer interaction applications are discussed.

Four Different Multimodal Setups for Non-Aerial Vehicle Simulations—A Case Study with a Speedboat Simulator

Vehicle simulators are multimodal interactive applications used in many human activities with different purposes. However, they are sometimes expensive systems that need to be carefully studied before they are designed and built, since some simulators can actually be much more expensive than the simulated vehicle. This is an important issue, although it is sometimes overlooked in scientific research. This paper proposes four different setups (with a variety of visual, sound, motion generation, and user-input interfaces) for non-aerial vehicle simulation, using a speedboat simulator as a case study. These setups are analysed in terms of their cost and their effectiveness is discussed. Rough figures are provided to give a comparative insight into the economic order of magnitude necessary to design and build a vehicle simulator.

Multimodal interaction with virtual worlds XMMVR: eXtensible language for MultiModal interaction with virtual reality worlds

Based on a philosophy of integrating components from multimodal interaction applications with 3D graphical environments, reusing already defined markup language for describing graphics, graphical and spoken interactions based on the interactive movie metaphor, a markup language for modeling scenes, behavior and interaction is sought. With the definition of this language, we hope to have a common framework for developing applications that allow multimodal interaction at 3D stages. Thus we have defined the basis of an architecture that allows us to integrate the components of such multimodal interaction applications in 3D virtual environments.

Special Issue on Haptics: Interfaces, Applications, and Perception

The last decade has seen significant advances in research on haptics and haptic interfaces. Device performance has improved, and the many commercial devices now available at reasonable prices indicate how haptic research will grow and new applications involving haptics will touch all aspects of daily life. Sophisticated systems require research beyond physical devices, such as modeling the physical properties of virtual objects, human physiology, and haptic evaluation. This special issue focuses on state-of-the-art design and development of haptic interfaces and explores potential applications of this technology and related issues such as tactile display, haptic rendering, physiology, and evaluation methodologies. The 15 papers were selected after a rigorous peer review from around the world and include diverse topics such as haptic device design and technology, tactile display and tactile sensing, collaborative multiuser haptic environments, haptic cognition, haptic rendering, tele-existence and multimodal interaction, and medical and rehabilitation applications. We thank the Editorial Board of JRM Journal for making this special issue possible. We also thank the authors for contributing their fine work and revising their papers for this issue, and extend our thanks and appreciation to the reviewers for their constructive comments and suggestions.

An approach integrating two complementary model-based environments for the construction of multimodal interactive applications

This paper presents a tool suite for the engineering of multimodal Post-WIMP Interactive Systems. The work presented here extends previous work done on design, prototyping, specification and verification of interactive systems and integrates two previously unrelated approaches. The first element of this integration is IC oM (a data-flow model dedicated to low-level input modelling) and its environment IC on which allows for editing and simulating IC oM models. The other element is ICOs (a formal description technique mainly dedicated to dialogue modelling) and its environment PetShop, which allows for editing, simulating and verifying ICOs models. This paper shows how these two approaches have been integrated and that this integration allows for engineering multimodal interactive systems. We show on a Range Slider case study how these tools can be used for prototyping interactive systems in general and multimodal interaction techniques in particular. We also present in details how the changes in the interaction techniques impact the models at various levels of the software architecture.

Interaction Builder: A Rapid Prototyping Tool for Developing Web-Based MMI Applications

We have developed Interaction Builder (IB), a rapid prototyping tool for constructing web-based Multi-Modal Interaction (MMI) applications. The goal of IB is making it easy to develop MMI applications with speech recognition, life-like agents, speech synthesis, web browsing, etc. For this purpose, IB supports the following interface and functions: (1) GUI for implementing MMI systems without the details of MMI and MMI description language, (2) functionalities of handling synchronized multimodal inputs/outputs, (3) a test run mode for run-time testing. The results of evaluation tests showed that the application development cycle using IB was significantly shortened in comparison with the time using a text editor both for MMI description language experts and for beginners.