Multi-modal haptic image recognition based on deep learning

Abstract

PurposeThis paper aims to improve the diversity and richness of haptic perception by recognizing multi-modal haptic images.Design/methodology/approachFirst, the multi-modal haptic data collected by BioTac sensors from different objects are pre-processed, and then combined into haptic images. Second, a multi-class and multi-label deep learning model is designed, which can simultaneously learn four haptic features (hardness, thermal conductivity, roughness and texture) from the haptic images, and recognize objects based on these features. The haptic images with different dimensions and modalities are provided for testing the recognition performance of this model.FindingsThe results imply that multi-modal data fusion has a better performance than single-modal data on tactile understanding, and the haptic images with larger dimension are conducive to more accurate haptic measurement.Practical implicationsThe proposed method has important potential application in unknown environment perception, dexterous grasping manipulation and other intelligent robotics domains.Originality/valueThis paper proposes a new deep learning model for extracting multiple haptic features and recognizing objects from multi-modal haptic images.