Mulsemedia data consolidation method

Abstract

The synchronization of multimodal data is one of the essential tasks related to mulse-media data processing. The concept of mulsemedia (MULtiple SEnsorial MEDIA) involves the registration, storage, processing, transmission and reproduction by computer-based tools of multimodal information about a physical object that humans can perceive through their senses. Such information includes audiovisual information (object's appearance, acoustic properties, etc.), tactile information (surface texture, temperature), kinesthetic information (weight, object's centre of gravity), information about its taste, smell, etc. The perception of mulsemedia information by a person is the process that exists over time. Because of this, the registration of mulsemedia data should be carried out with the fixation of the moments of time when the relevant mulsemedia information existed or its perception made sense for a human who supervises the object as mulsemedia data is temporal. This paper presents a method that enables the consolidation and synchronization of mulsemedia data using the principles of multithreading. The universal method was designed to support combining data of different modalities in parallel threads. The application of the proposed method solves problems associated with integrating data of different modalities and formats in the same time interval. The effectiveness of applying this method increases by us-ing multithreaded distributed computing. This method is designed for use in the development of mulsemedia software systems. The modified JSON format (TJSON – Timeline JSON) was proposed in the paper, as well. TJSON-object is a complex data structure for representing the synchronized mulsemedia data and their further processing. The proposed method can be further extended with other approaches and technologies. For example, artificial intelligence methods can be applied to assess the correlation between data from different modalities. This can help improve the method's accuracy and the output files' quality.