Systematic Review of Multimodal Human–Computer Interaction

Jose Daniel Azofeifa,José Martín Molina-Espinosa,Julieta Noguez,Alejandra J Magana,Bedrich Benes,Sergio Ruiz

doi:10.3390/informatics9010013

Jose Daniel Azofeifa, José Martín Molina-Espinosa + Show 4 more

Open Access

https://doi.org/10.3390/informatics9010013

Copy DOI

Abstract

This document presents a systematic review of Multimodal Human–Computer Interaction. It shows how different types of interaction technologies (virtual reality (VR) and augmented reality, force and vibration feedback devices (haptics), and tracking) are used in different domains (concepts, medicine, physics, human factors/user experience design, transportation, cultural heritage, and industry). A systematic literature search was conducted identifying 406 articles initially. From these articles, we selected 112 research works that we consider most relevant for the content of this article. The articles were analyzed in-depth from the viewpoint of temporal patterns, frequency of usage in types of technology in different domains, and cluster analysis. The analysis allowed us to answer relevant questions in searching for the next steps in work related to multimodal HCI. We looked at the typical technology type, how the technology type and frequency have changed in time over each domain, and how papers are grouped across metrics given their similarities. This analysis determined that VR and haptics are the most widely used in all domains. While VR is the most used, haptic interaction is presented in an increasing number of applications, suggesting future work on applications that configure VR and haptic together.

Highlights

We present a systematic review of multimodal human-computer interaction (HCI) with the primary objective of showing how different types of technologies are used in different subject areas, called domains
In order to guide our study towards the objective of determining the effort made in multimodal interaction and the steps in multimodal HCI work, we identified the following four research questions: RQ1: How has the type of technology changed over time in each domain? RQ2: What is the typical technology type by domain? RQ3: How has the frequency of research publications changed over time by domain? RQ4: How are research publications grouped across metrics given their similarities?
While the same authors follow the “bidirectional communication theory” approach by Marko [133], we focus on the link of multimodal HCI technology in the particular domains of Concepts and overviews, Medicine, Physics, Transportation, Cultural heritage, and Industry, in addition to User experience design (UX), in order to highlight specific technology configurations that are typical for each domain, suggesting future lines of work from the characteristics of these configurations

Summary

Introduction

We present a systematic review of multimodal human-computer interaction (HCI) with the primary objective of showing how different types of technologies are used in different subject areas, called domains. We are not aware of a systematic review covering multiple technologies used in different application areas. Most of the above-mentioned studies analyze, in-depth, a small number of domains and types of technology (e.g., VR in manufacturing, haptics in medicine, etc.). There is a need for a more extensive study covering various domains that does not provide a detailed review but paints a larger horizontal picture. A critical research endeavor would be to identify the combination of multiple domains and various technologies for the development and research of HCI.

Objectives

Methods

Results

Conclusion