Evaluating Integration Strategies for Visuo-Haptic Object Recognition

Sibel Toprak,Stefan Wermter,Nicolás Navarro-Guerrero

doi:10.1007/s12559-017-9536-7

Abstract

In computational systems for visuo-haptic object recognition, vision and haptics are often modeled as separate processes. But this is far from what really happens in the human brain, where cross- as well as multimodal interactions take place between the two sensory modalities. Generally, three main principles can be identified as underlying the processing of the visual and haptic object-related stimuli in the brain: (1) hierarchical processing, (2) the divergence of the processing onto substreams for object shape and material perception, and (3) the experience-driven self-organization of the integratory neural circuits. The question arises whether an object recognition system can benefit in terms of performance from adopting these brain-inspired processing principles for the integration of the visual and haptic inputs. To address this, we compare the integration strategy that incorporates all three principles to the two commonly used integration strategies in the literature. We collected data with a NAO robot enhanced with inexpensive contact microphones as tactile sensors. The results of our experiments involving every-day objects indicate that (1) the contact microphones are a good alternative to capturing tactile information and that (2) organizing the processing of the visual and haptic inputs hierarchically and in two pre-processing streams is helpful performance-wise. Nevertheless, further research is needed to effectively quantify the role of each identified principle by itself as well as in combination with others.

Highlights

Most envisioned applications require robots to recognize objects well
We examine the three integration strategies illustrated in Fig. 3: (a) The monolithic integration strategy, which is an instance of the pre-mapping fusion approach and performs the classification on the concatenation of all object descriptors, (b) the modality-based integration strategy, which is used the most often and where the visual and haptic inputs are processed in two separate streams before the results of the pre-processing are integrated in the final object label predictor, and (c) the brain-inspired integration strategy, which incorporates all of the abovementioned processing principles
This motivated us to take on the research question of whether the object recognition performance of artificial systems can be improved by incorporating the processing principles that are involved in the integration of the visual and haptic object-related information in the brain

Summary

Introduction

Most envisioned applications require robots to recognize objects well. vision is important for object recognition, there are certain hard-to-overcome challenges when relying on it alone [20]. The human brain, deals with this information differently: interactions between vision and touch take place in the cortex [39]. These interactions can be crossmodal, meaning that the haptic stimuli activate regions traditionally believed to be visual, or multimodal, in which case the visual and the haptic stimuli converge. Incorporating these insights regarding how the human brain combines vision and haptics to recognize objects might help robots approach human proficiency, eventually. The objectives of this paper are the following: 1. to present a functional description of how visuo-haptic object recognition is performed in the brain and derive organizational principles that could be used for robots (see “Visuo-Haptic Principles of the Human Brain”)

Objectives

Methods

Results

Discussion

Conclusion