3D force identification and prediction using deep learning based on a Gelsight-structured sensor

Abstract

The mechanics perception of human surface cells is very important for humans to grasp objects. However, most of the mainstream sensors are based on electronic principles, and such tactile sensors are easily corroded and susceptible to electromagnetic environment interference. In this work, we fabricate a soft tactile sensor based on the structure of Gelsight for three-dimensional force classification and measurement. Gelsight can show the shape change of the contact surface, which can be used for identifying the type and value of the force. Meanwhile, a deep learning network is employed to analyze the data and output the type and value of the undertest force. Our sensor can measure the torsion angle of the contact surface, which is a physical quantity not measured by the previous Gelsight sensor. The sensor combines deep learning to realize contact surface segmentation and contact force prediction. Numerous experiments demonstrate that the neural network assisted Gelsight-structured sensor not only has a compact structure, but also can accurately distinguish the types of force and estimate the mechanical value with a resolution of 0.2 N in a wide measurement range. To the best of our knowledge, this is the first time that a Gelsight-structured sensor can distinguish torsion angle with a resolution of 3°. All the results can be automatically output, which paves the way of practical applications.