The contribution of early multimodal data fusion for subjectivity in HCI

Abstract

Listen

Translate article

Information fusion methodologies are one of the most relevant issues at the core of multimodal systems. More than the simple juxtapositions of various sensory modalities at the interface, their possible synergistic combinations should be considered, in order to overcome individual weaknesses and obtain the sum of their expressive forces. The current paradigm in the association of data in interactive processes of a multimodal nature proposes the existence of three relevant aggregating levels: the level of fusion of characteristics - early fusion; the level of fusion of semantics - late fusion; and a hybrid fusion, that implies the simultaneous use of the fusion levels of characteristics and semantics. On the part of most top authors, there is a frequent disregard for the potential of the association of raw signals, immediately after their acquisition - the so-called (early fusion), notably when coming from different modalities or input channels. This article proposes to clarify, through literature review, the importance of this type of fusion for the integration of subjectivity in the scope of artistic explorations, so that one can consider combinations of creative scope that contemplate error as a form of communication and expression.