Optical Tactile Sensor Research Articles

Thiol-ene UV-curable sponge electrolyte for low-voltage color changing wearable tactile device

Interest in developing advanced flexible materials that are stable, mechanically deformable, lightweight, cost-effective, and eco-friendly is ever-growing to support high-performance state-of-the-art wearable electronic devices. Another critical requirement of wearable devices is low energy consumption during operation for prolonged use. Herein, we demonstrate a compressible electrochromic device (ECD) using the three-dimensional (3D) compressible sponge electrolyte layer. A 3D porous sponge was fabricated using a facile and eco-friendly method of leaching sugar, a pore-creating agent, from an UV-cured skeletal structure. The simple structure of a compressible ECD is advantageous for its applicability as an optical tactile sensor device. A low-voltage operation, as evidenced by optical measurements, can ensure the long-term performance of the sensor. Furthermore, the color change in the ECD that occurs with the applied pressure acts as an effective pressure-sensing mechanism for the system involved. Thus, compressible sponge electrolytes layers are promising for various wearable tactile device applications.

Goal-Driven Robotic Pushing Using Tactile and Proprioceptive Feedback

In robots, nonprehensile manipulation operations such as pushing are a useful way of moving large, heavy, or unwieldy objects, moving multiple objects at once, or reducing uncertainty in the location or pose of objects. In this study, we propose a reactive and adaptive method for robotic pushing that uses rich feedback from a high-resolution optical tactile sensor to control push movements instead of relying on analytical or data-driven models of push interactions. Specifically, we use goal-driven tactile exploration to actively search for stable pushing configurations that cause the object to maintain its pose relative to the pusher while incrementally moving the pusher and object toward the target. We evaluate our method by pushing objects across planar and curved surfaces. For planar surfaces, we show that the method is accurate and robust to variations in initial contact position/angle, object shape, and start position; for curved surfaces, the performance is degraded slightly. An immediate consequence of our work is that it shows that explicit models of push interactions might be sufficient but are not necessary for this type of task. It also raises the interesting question of which aspects of the system should be modeled to achieve the best performance and generalization across a wide range of scenarios. Finally, it highlights the importance of testing on nonplanar surfaces and in other more complex environments when developing new methods for robotic pushing.

A liquid lens-based optical sensor for tactile sensing

Tactile sensing plays a crucial role in robot manipulation, robot interaction, and health monitoring. Because of high sensitivity, simple structure, and superior interference immunity, optical tactile sensors based on optical imaging or optical conduction have been one of the most active research. Herein, a novel liquid lens-based optical sensor (LLOS) is presented. Different with existed optical tactile sensors, the main body of the proposed sensor belongs to a variable-focus optical lens with a liquid-membrane structure, and its focal length is changed with the contact force, thereby changing the propagation direction of light and affecting the perceived light intensity of the photosensitive element. By conducting some testing experiments, the LLOS demonstrates fast response (about 0.021 s), stable dynamic response characteristics, and good linearity (R-squared is about 0.99), repeated measurement accuracy (<0.006 V), and measurement accuracy (<0.2 N). Hence, the LLOS provides a new and promising method to measure tactile and has potential application in robotics nondestructive grasping and interactive input devices.

Pose-Based Tactile Servoing: Controlled Soft Touch Using Deep Learning

This article describes a new way of controlling robots using soft tactile sensors: pose-based tactile servo (PBTS) control. The basic idea is to embed a tactile perception model for estimating the sensor pose within a servo control loop that is applied to local object features, such as edges and surfaces. PBTS control is implemented with a soft, curved optical tactile sensor [the Bristol Robotics Laboratory (BRL) TacTip] using a convolutional neural network trained to be insensitive to shear. As a consequence, robust and accurate controlled motion over various complex 3D objects is attained.

Soft Biomimetic Optical Tactile Sensing With the TacTip: A Review

Reproducing the capabilities of the human sense of touch in machines is an important step in enabling robot manipulation to have the ease of human dexterity. A combination of robotic technologies will be needed, including soft robotics, biomimetics and the high-resolution sensing offered by optical tactile sensors. This combination is considered here as a SoftBOT (Soft Biomimetic Optical Tactile) sensor. This article reviews the BRL TacTip as a prototypical example of such a sensor. Topics include the relation between artificial skin morphology and the transduction principles of human touch, the nature and benefits of tactile shear sensing, 3D printing for fabrication and integration into robot hands, the application of AI to tactile perception and control, and the recent step-change in capabilities due to deep learning. This review consolidates those advances from the past decade to indicate a path for robots to reach human-like dexterity.

A Robust Controller for Stable 3D Pinching Using Tactile Sensing

This letter proposes a controller for stable grasping of unknown-shaped objects by two robotic fingers with tactile fingertips. The grasp is stabilised by rolling the fingertips on the contact surface and applying a desired grasping force to reach an equilibrium state. The validation is both in simulation and on a fully-actuated robot hand (the Shadow Modular Grasper) fitted with custom-built optical tactile sensors (based on the BRL TacTip). The controller requires the orientations of the contact surfaces, which are estimated by regressing a deep convolutional neural network over the tactile images. Overall, the grasp system is demonstrated to achieve stable equilibrium poses on various objects ranging in shape and softness, with the system being robust to perturbations and measurement errors. This approach also has promise to extend beyond grasping to stable in-hand object manipulation with multiple fingers.

Fiber optic tactile sensor for surface roughness recognition by machine learning algorithms

In this study, a sensor tip with a metallic hemispherical nozzle tip (MHNT) design based on the Fabry-Perot interferometer was developed for surface roughness recognition (SRR). Sandpaper samples with ten different arithmetical mean deviations of the surface (Sa) values were used as surfaces to be recognized. The feature vectors were found by applying the discrete wavelet transform (DWT) to the analog signals obtained from the sandpaper samples. Machine learning (ML) algorithms K-nearest neighbor (KNN) and support vector machine (SVM) were used for classification. An in-depth recognition process was carried out using the classifiers’ different length criteria and kernel types. In the test process, each category consists of two sub-categories as testing within the training dataset (TWITD) and testing without the training dataset (TWOTD). The experiments were carried out in a controlled manner with the conveyor belt system (CBS) and manual. As a result of the experimental studies, the average recognition rates (Rave) for CBS were found as 84.2% and 81.6% for TWITD and TWOTD, while the Rave for the manual are found as 80% and 77.5% for TWITD and TWOTD, respectively.

Tactile Sensing Systems for Tumor Characterization: A Review

This paper presents a review of Tactile Sensing Systems, with an emphasis on their application to the non-invasive characterization of tumors. Emulating the perceptual mechanism of the human skin, the Tactile Sensing Systems characterize tumors using touch sensors by quantifying the mechanical properties such as size and elasticity. The authors survey several tactile transduction methods: capacitive, piezoresistive, piezoelectric, magnetic, and optical. The advantages and disadvantages of different tactile sensors are discussed. The complex human sense of touch is emulated using tactile sensor data, novel data processing algorithms, and near real-time interpretation in a human-readable format. Tactile Sensing Systems utilize tactile sensors and other subsystems to come up with accurate mechanical properties of the touched objects. We review Elasticity Determination Systems, which are a special case of Tactile Sensing Systems. These systems are based on capacitive sensors, piezoelectric sensors, elastography, and optical tactile sensors. Then the optical Tactile Sensing System is discussed in detail; architecture, sensing principle, and algorithms to compute a risk score. Moreover, a survey of multimodal Tactile Sensing Systems, which broaden the capabilities of existing tactile sensing systems, is presented. The paper concludes with discussions and future research directions.

Transfer of Learning from Vision to Touch: A Hybrid Deep Convolutional Neural Network for Visuo-Tactile 3D Object Recognition.

Transfer of learning or leveraging a pre-trained network and fine-tuning it to perform new tasks has been successfully applied in a variety of machine intelligence fields, including computer vision, natural language processing and audio/speech recognition. Drawing inspiration from neuroscience research that suggests that both visual and tactile stimuli rouse similar neural networks in the human brain, in this work, we explore the idea of transferring learning from vision to touch in the context of 3D object recognition. In particular, deep convolutional neural networks (CNN) pre-trained on visual images are adapted and evaluated for the classification of tactile data sets. To do so, we ran experiments with five different pre-trained CNN architectures and on five different datasets acquired with different technologies of tactile sensors including BathTip, Gelsight, force-sensing resistor (FSR) array, a high-resolution virtual FSR sensor, and tactile sensors on the Barrett robotic hand. The results obtained confirm the transferability of learning from vision to touch to interpret 3D models. Due to its higher resolution, tactile data from optical tactile sensors was demonstrated to achieve higher classification rates based on visual features compared to other technologies relying on pressure measurements. Further analysis of the weight updates in the convolutional layer is performed to measure the similarity between visual and tactile features for each technology of tactile sensing. Comparing the weight updates in different convolutional layers suggests that by updating a few convolutional layers of a pre-trained CNN on visual data, it can be efficiently used to classify tactile data. Accordingly, we propose a hybrid architecture performing both visual and tactile 3D object recognition with a MobileNetV2 backbone. MobileNetV2 is chosen due to its smaller size and thus its capability to be implemented on mobile devices, such that the network can classify both visual and tactile data. An accuracy of 100% for visual and 77.63% for tactile data are achieved by the proposed architecture.

Optimal Deep Learning for Robot Touch: Training Accurate Pose Models of 3D Surfaces and Edges

This article illustrates the application of deep learning to robot touch by considering a basic yet fundamental capability: estimating the relative pose of part of an object in contact with a tactile sensor. We begin by surveying deep learning applied to tactile robotics, focusing on optical tactile sensors, which help to link touch and deep learning for vision. We then show how deep learning can be used to train accurate pose models of 3D surfaces and edges that are insensitive to nuisance variables, such as motion-dependent shear. This involves including representative motions as unlabeled perturbations of the training data and using Bayesian optimization of the network and training hyperparameters to find the most accurate models. Accurate estimation of the pose from touch will enable robots to safely and precisely control their physical interactions, facilitating a wide range of object exploration and manipulation tasks.

Shear, Torsion and Pressure Tactile Sensor via Plastic Optofiber Guided Imaging

Object manipulation performed by robots refers to the art of controlling the shape and location of an object through force constraints with robot end-effectors, both robot hands, and grippers. The success of task execution is usually guaranteed by the sense of touch. In this work, we present an optical tactile sensor incorporating plastic optical fibers, transparent silicone rubber, and an off-the-shelf color camera that can detect: translational and rotational shear forces, and contact location and its normal force. Contact localization is possible thanks to the shear strain. Specifically, one of the layers stretches so that its thickness decreases. The decrease in the thickness results in the color change at the point of contact. Elastic behavior of the sensing media provides a robust rotational and translational shear detection mechanism when torque and planar force, respectively, are applied onto the sensing surface. Thanks to the plastic optofibers, signal processing electronics are placed away from the sensing surface making the sensor immune to hazardous environments. Machine learning techniques were used to benchmark the sensing performance of the sensor. By implementing a multi-output CNN model, the contact type was classified into normal and shear or torsional deformation and their corresponding continuous contact features were estimated.

Miniaturized Optical Force Sensor for Minimally Invasive Surgery With Learning-Based Nonlinear Calibration

A simple and miniaturized optical tactile sensor for integrating with robotic and manual minimally invasive surgery graspers is proposed in this study. For better miniaturization, the sensing principle of constant-bending-radius light intensity modulation was replaced with a variable-bending-radius modulation principle, and the pertinent theoretical formulation was derived. Afterward, a finite element model of the sensor was optimized using response surface optimization technique. The optimized sensor design was 14.0 mm long, 1.8 mm wide and 4 mm high. Next, the sensor was prototyped using SLA 3D printing technique. Also, the sensor was calibrated using a rate-dependent learning-based support-vector-regression algorithm. Calibration was 96% linear with a goodness-of-fit of 93% and mean absolute error of 0.085±0.096 N. Furthermore, the sensor was tested under cyclic triangular compression with a 3 sec pause between loading and unloading as well as manual grasping. Mean absolute error of 0.12±0.08 N, the minimum force of 0.14 N, and repeatability of 0.07 N showed the acceptable performance of the proposed sensor for surgical applications. Moreover, the sensor showed the capability of working under combined dynamic and static loading conditions with low hysteresis, i.e., 0.057 N/cycle.

Creating a Soft Tactile Skin Employing Fluorescence Based Optical Sensing

Currently, optical tactile sensors propose solutions to measure contact forces at the tip of flexible medical instruments. However, the sensing capability of normal pressures applied to the surface along the tool body is still an open challenge. To deal with this challenge, this letter proposes a sensor design employing an angled tip optical fiber to measure the intensity modulation of a fluorescence signal proportional to the applied force. The fiber is used as both emitter of the excitation light and receiver of the fluorescence signal. This configuration allows to (i) halve the number of optical fibers and (ii) improve the signal to noise ratio thanks to the wavelength shift between excitation and fluorescence emission. The proposed design makes use of soft and flexible materials only, avoiding the size constraints given by rigid optical components and facilitating further miniaturization. The employed materials are bio-compatible and guarantee chemical inertness and non-toxicity for medical uses. In this work, the sensing principle is validated using a single optical fiber. Then, a soft stretchable skin pad, containing four tactile sensing elements, is presented to demonstrate the feasibility of this new force sensor design.

Efficient and Small Network Using Multi-Trim Network Structure for Tactile Object Recognition on Embedded Systems

Tactile object recognition (TOR) is critical in robot perception. However, as an embedded system, a robot brain has a fixed resource budget and is unsuitable for modern convolutional neural networks (CNNs). To bridge this gap, we present a simple network-compression approach that improves the accuracy-latency trade-off of the network. The multi-trim network structure (MTNS) is a robust combination of network compression (NC) techniques providing a lightweight network with no performance drop. Furthermore, as an optical tactile sensor, we present a random-dot sensor that obtains rich information with a single touch, thus avoiding modality fusion. The random-dot sensor captures the object shapes and inputs them to TOR. In an experimental evaluation, we compare the performances of the proposed MTNS approach with those of CNN filter pruning, the network quantization technique, an adaptive mixture of low-rank factorizations, and knowledge distillation. The MTNS better resolved the accuracy-latency trade-off in tactile object recognition than the modern NC methods. By combining the random-dot sensor and MTNS approach, TOR enhances the accuracy and processing time performances.

From Pixels to Percepts: Highly Robust Edge Perception and Contour Following Using Deep Learning and an Optical Biomimetic Tactile Sensor

Deep learning has the potential to have the impact on robot touch that it has had on robot vision. Optical tactile sensors act as a bridge between the subjects by allowing techniques from vision to be applied to touch. In this paper, we apply deep learning to an optical biomimetic tactile sensor, the TacTip, which images an array of papillae (pins) inside its sensing surface analogous to structures within human skin. Our main result is that the application of a deep CNN can give reliable edge perception and thus a robust policy for planning contact points to move around object contours. Robustness is demonstrated over several irregular and compliant objects with both tapping and continuous sliding, using a model trained only by tapping onto a disk. These results relied on using techniques to encourage generalization to tasks beyond which the model was trained. We expect this is a generic problem in practical applications of tactile sensing that deep learning will solve. A video demonstrating the approach can be found at https://www.youtube.com/watch?v=QHrGsG9AHts

Design, Motivation and Evaluation of a Full-Resolution Optical Tactile Sensor.

Human skin is capable of sensing various types of forces with high resolution and accuracy. The development of an artificial sense of touch needs to address these properties, while retaining scalability to large surfaces with arbitrary shapes. The vision-based tactile sensor proposed in this article exploits the extremely high resolution of modern image sensors to reconstruct the normal force distribution applied to a soft material, whose deformation is observed on the camera images. By embedding a random pattern within the material, the full resolution of the camera can be exploited. The design and the motivation of the proposed approach are discussed with respect to a simplified elasticity model. An artificial deep neural network is trained on experimental data to perform the tactile sensing task with high accuracy for a specific indenter, and with a spatial resolution and a sensing range comparable to the human fingertip.

Tactile Sensor for Manipulation of Deformable Object

The variable physical property of deformable objects, which are very flexible, soft and viscoelastic, causes the design of reliable automated handling system relatively difficult. In fact, most of these objects tend to be handled manually during the handling process. Therefore, a new optical tactile sensor for an intelligent handling of the non-rigid materials is presented in this paper. Mathematical modelling and control algorithm are developed and the tactile sensor is calibrated in this research. Based on the results that have been recorded, the surface characterization with the respect to normal force applied to the object is attained. A gripper handling system is used to accommodate variable physical properties of the deformable materials, which are very flexible, soft and viscoelastic. In addition to that, the gripper needs to handle the materials with the minimum deformation so that less distortion, and higher accuracy of manipulation can be achieved. Efficient and accurate modelling of deformations is crucial for grasping analysis.

Graphene-based optical waveguide tactile sensor for dynamic response

Optical tactile sensors based on a directional coupler have been widely investigated because of their many advantages. However, one important requirement limits their wide application: the refractive index of the upper superstrate must be equal to or larger than that of the optical waveguide core. To overcome this disadvantage, an optical waveguide tactile sensor using graphene is proposed and its operational feasibility was validated experimentally. The pressure-dependent lateral deformation of the low-index prism-like microstructure on an elastomer superstrate has a key role in optically measuring the mechanical pressure. By mechanically varying the lateral deformation area, the waveguide core-graphene-polydimethylsiloxane (PDMS) interface area was adjusted and the amount of light absorption by graphene became tunable, even when the refractive index of the superstrate was lower than that of the waveguide core. The dynamic response of the sensor was accurately matched to the repeated pressing and release time of the pressure, and exhibited a real-time response to multi-stepped mechanical pressing and releasing using a piezoelectric motor. The proposed graphene-based optical tactile sensor is foundational to the use of a wide range of materials for overcoming the shortcoming of a directional coupler-based optical tactile sensor.

A High-Sensitivity Tactile Sensor Array Based on Fiber Bragg Grating Sensing for Tissue Palpation in Minimally Invasive Surgery

This paper presents a novel and high-sensitivity optical tactile sensor array based on fiber Bragg grating (FBG) to explore and localize tissue abnormalities during tissue palpation in a minimally invasive surgery. This sensor consists of five separate tactile cells with the same configuration, associated fixations, and holder parts. Each tactile cell is mainly composed of a spiral elastomer, a suspended optical fiber inscribed with an FBG element, and a contact head connecting the elastomer with a threaded connection. The two ends of the fiber were glued on the two sides of the spiral elastomer to form a tight suspension status. This configuration can effectively avoid the FBG chirping failure and achieve a higher sensitivity compared with the conventional direct FBG-pasting methods. The corresponding theoretical model has been derived, and the finite element method based simulation has been performed to facilitate the structure design and investigate both static and dynamic performances. Both static calibration experiments and dynamic loading experiments have been conducted to characterize the sensor's performance. The prototyped sensor can achieve a high resolution of up to 0.93 mN and is able to operate within a force measurement range of 0–5 N with a maximum relative error of less than 8.22%. The further surface estimation experiments and in-vitro palpation implementation on a silicone phantom embedded with simulated tumors have validated the effectiveness of the proposed design. The in-vitro experiments have demonstrated an excellent capacity to detect tumors with an embedded depth of up to 8 mm.

A 3-D Surface Reconstruction with Shadow Processing for Optical Tactile Sensors.

An optical tactile sensor technique with 3-dimension (3-D) surface reconstruction is proposed for robotic fingers. The hardware of the tactile sensor consists of a surface deformation sensing layer, an image sensor and four individually controlled flashing light emitting diodes (LEDs). The image sensor records the deformation images when the robotic finger touches an object. For each object, four deformation images are taken with the LEDs providing different illumination directions. Before the 3-D reconstruction, the look-up tables are built to map the intensity distribution to the image gradient data. The possible image shadow will be detected and amended. Then the 3-D depth distribution of the object surface can be reconstructed from the 2-D gradient obtained using the look-up tables. The architecture of the tactile sensor and the proposed signal processing flow have been presented in details. A prototype tactile sensor has been built. Both the simulation and experimental results have validated the effectiveness of the proposed 3-D surface reconstruction method for the optical tactile sensors. The proposed 3-D surface reconstruction method has the unique feature of image shadow detection and compensation, which differentiates itself from those in the literature.