Abstract
Technology developments have expanded the diversity of interaction modalities that can be used by an agent (either a human or machine) to interact with a computer system. This expansion has created the need for more natural and user-friendly interfaces in order to achieve effective user experience and usability. More than one modality can be provided to an agent for interaction with a system to accomplish this goal, which is referred to as a multimodal interaction (MI) system. The Internet of Things (IoT) and augmented reality (AR) are popular technologies that allow interaction systems to combine the real-world context of the agent and immersive AR content. However, although MI systems have been extensively studied, there are only several studies that reviewed MI systems that used IoT and AR. Therefore, this paper presents an in-depth review of studies that proposed various MI systems utilizing IoT and AR. A total of 23 studies were identified and analyzed through a rigorous systematic literature review protocol. The results of our analysis of MI system architectures, the relationship between system components, input/output interaction modalities, and open research challenges are presented and discussed to summarize the findings and identify future research and development avenues for researchers and MI developers.
Highlights
Our analysis revealed that multimodal interaction (MI) systems have been studied from various perspectives
Which Modalities Are Used in MI Systems with Internet of Things (IoT) and augmented reality (AR)?
We examined the titles and abstracts in order to identify whether the studies presented practical or theoretical MI systems that consisted of IoT and AR/mixed reality (MR)
Summary
Since the first personal computers were released in the late 1970s, computer graphics, networking, and data processing technologies have evolved to provide high accuracy and powerful performance for rich interactive applications. These developments have enabled various methods of interaction between humans and machines to provide more natural and user-friendly interfaces, thereby contributing to improve user experience and usability [1]. “State” refers to the contextual information of the sender, such as temperature, pressure, and other information Affective computing is another way to understand the sender’s context, which is a discipline that focuses on the identification of a human’s emotional state via computer and sensor devices [23]. Predicting the sender’s intent by an MI system with high accuracy is an open research problem in MI system development [6]
Sign-in/Register to view highlights & summary 
Published Version (
Free)
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call