Tactile Data Research Articles

Tactile Sensing Systems for Tumor Characterization: A Review

This paper presents a review of Tactile Sensing Systems, with an emphasis on their application to the non-invasive characterization of tumors. Emulating the perceptual mechanism of the human skin, the Tactile Sensing Systems characterize tumors using touch sensors by quantifying the mechanical properties such as size and elasticity. The authors survey several tactile transduction methods: capacitive, piezoresistive, piezoelectric, magnetic, and optical. The advantages and disadvantages of different tactile sensors are discussed. The complex human sense of touch is emulated using tactile sensor data, novel data processing algorithms, and near real-time interpretation in a human-readable format. Tactile Sensing Systems utilize tactile sensors and other subsystems to come up with accurate mechanical properties of the touched objects. We review Elasticity Determination Systems, which are a special case of Tactile Sensing Systems. These systems are based on capacitive sensors, piezoelectric sensors, elastography, and optical tactile sensors. Then the optical Tactile Sensing System is discussed in detail; architecture, sensing principle, and algorithms to compute a risk score. Moreover, a survey of multimodal Tactile Sensing Systems, which broaden the capabilities of existing tactile sensing systems, is presented. The paper concludes with discussions and future research directions.

Measuring the sensitivity of tactile temporal order judgments in sighted and blind participants using the adaptive psi method.

Spatial locations of somatosensory stimuli are coded according to somatotopic (anatomical distribution of the sensory receptors on the skin surface) and spatiotopic (position of the body parts in external space) reference frames. This was mostly evidenced by means of temporal order judgment (TOJ) tasks in which participants discriminate the temporal order of two tactile stimuli, one applied on each hand. Because crossing the hands generates a conflict between anatomical and spatial responses, TOJ performance is decreased in such posture, except for congenitally blind people, suggesting a role of visual experience in somatosensory perception. In previous TOJ studies, stimuli were generally presented using the method of constant stimuli-that is, the repetition of a predefined sample of stimulus-onset asynchronies (SOA) separating the two stimuli. This method has the disadvantage that a large number of trials is needed to obtain reliable data when aiming at dissociating performances of groups characterized by different cognitive abilities. Indeed, each SOA among a large variety of different SOAs should be presented the same number of times irrespective of the participant's performance. This study aimed to replicate previous tactile TOJ data in sighted and blind participants with the adaptive psi method in order to validate a novel method that adapts the presented SOA according to the participant's performance. This allows to precisely estimate the temporal sensitivity of each participant while the presented stimuli are adapted to the participant's individual discrimination threshold. We successfully replicated previous findings in both sighted and blind participants, corroborating previous data using a more suitable psychophysical tool.

Generative Partial Visual-Tactile Fused Object Clustering

Visual-tactile fused sensing for object clustering has achieved significant progresses recently, since the involvement of tactile modality can effectively improve clustering performance. However, the missing data (i.e., partial data) issues always happen due to occlusion and noises during the data collecting process. This issue is not well solved by most existing partial multi-view clustering methods for the heterogeneous modality challenge. Naively employing these methods would inevitably induce a negative effect and further hurt the performance. To solve the mentioned challenges, we propose a Generative Partial Visual-Tactile Fused (i.e., GPVTF) framework for object clustering. More specifically, we first do partial visual and tactile features extraction from the partial visual and tactile data, respectively, and encode the extracted features in modality-specific feature subspaces. A conditional cross-modal clustering generative adversarial network is then developed to synthesize one modality conditioning on the other modality, which can compensate missing samples and align the visual and tactile modalities naturally by adversarial learning. To the end, two pseudo-label based KL-divergence losses are employed to update the corresponding modality-specific encoders. Extensive comparative experiments on three public visual-tactile datasets prove the effectiveness of our method.

How to Select and Use Tools? : Active Perception of Target Objects Using Multimodal Deep Learning

Selection of appropriate tools and use of them when performing daily tasks is a critical function for introducing robots for domestic applications. In previous studies, however, adaptability to target objects was limited, making it difficult to accordingly change tools and adjust actions. To manipulate various objects with tools, robots must both understand tool functions and recognize object characteristics to discern a tool-object-action relation. We focus on active perception using multimodal sensorimotor data while a robot interacts with objects, and allow the robot to recognize their extrinsic and intrinsic characteristics. We construct a deep neural networks (DNN) model that learns to recognize object characteristics, acquires tool-object-action relations, and generates motions for tool selection and handling. As an example tool-use situation, the robot performs an ingredients transfer task, using a turner or ladle to transfer an ingredient from a pot to a bowl. The results confirm that the robot recognizes object characteristics and servings even when the target ingredients are unknown. We also examine the contributions of images, force, and tactile data and show that learning a variety of multimodal information results in rich perception for tool use.

Evaluation of efficiency of Database Systems Used for IoT Applications-Review

The data put away in IoT databases increments as the IoT applications stretch out all through smart city appliances, industry, and agriculture. New database systems should handle tremendous measures of tactile and actuator data progressively or intuitively. Confronting this first rush of IoT unrest, database management step by step to acquire a piece of the overall industry, grow new capacities and endeavor to defeat past discharges' weaknesses while giving highlights to the IoT. There are two well-known database types: The Relational Database Management Systems and NoSQL databases, with NoSQL making progress on IoT data storage. With regards to this paper, these two sorts are inspected. Zeroing in on open source databases, the creators investigate IoT data sets and represent an answer to the inquiry that one performs better than the next.

Visual Affordance Guided Tactile Material Recognition for Waste Recycling

Because more and more solid waste is generated, in particular, in cities, the management of solid waste disposal has become a global challenge. A solution is to find an effective way to sort solid waste materials and recycle them into reusable products. In this article, we propose to use a material recognition method with vision-guided tactile to form a robotic system for waste sorting. The vision guidance module integrates an object detector and an affordance network together. It allows the robot to not only detect the desired containers and packaging from an assortment of the waste but also obtain a configuration to grasp the target and actively collect its tactile data. By classifying the object with the tactile data, the robot can sort containers and packaging into their respective categories according to the type of material. Our experimental results demonstrate the effectiveness of the proposed robotic waste sorting system in sorting containers and packaging various types of materials. Note to Practitioners—The management of waste has become a great challenge in environmental protection in the world. We propose a robotic waste sorting system, which utilizes a vision-guided tactile sensing approach to find target waste and sort the waste according to materials. A visual module is used to find the waste of interest. A class-specific affordance map is generated to guide a robotic hand to actively grasp waste and collect the tactile data for material recognition. With the recorded tactile data, the robot can recognize the material of the target waste and sort the waste according to the type of material. The proposed system demonstrates good performance in waste sorting, and it can be easily implemented in practical scenarios. The target material can be generalized to a broader group of objects.

Recent trends and role of large area flexible electronics in shape sensing application – a review

PurposeThis paper aims to review the shape sensing techniques using large area flexible electronics (LAFE). Shape perception of humanoid robots using tactile data is mainly focused.Design/methodology/approachResearch papers on different shape sensing methodologies of objects with large area, published in the past 15 years, are reviewed with emphasis on contact-based shape sensors. Fiber optics based shape sensing methodology is discussed for comparison purpose.FindingsLAFE-based shape sensors of humanoid robots incorporating advanced computational data handling techniques such as neural networks and machine learning (ML) algorithms are observed to give results with best resolution in 3D shape reconstruction.Research limitations/implicationsThe literature review is limited to shape sensing application either two- or three-dimensional (3D) LAFE. Optical shape sensing is briefly discussed which is widely used for small area. Optical scanners provide the best 3D shape reconstruction in the noncontact-based shape sensing; here this paper focuses only on contact-based shape sensing.Practical implicationsContact-based shape sensing using polymer nanocomposites is a very economical solution as compared to optical 3D scanners. Although optical 3D scanners can provide a high resolution and fast scan of the 3D shape of the object, they require line of sight and complex image reconstruction algorithms. Using LAFE larger objects can be scanned with ML and basic electronic circuitory, which reduces the price hugely.Social implicationsLAFE can be used as a wearable sensor to monitor critical biological parameters. They can be used to detect shape of large body parts and aid in designing prosthetic devices. Tactile sensing in humanoid robots is accomplished by electronic skin of the robot which is a prime example of human–machine interface at workplace.Originality/valueThis paper reviews a unique feature of LAFE in shape sensing of large area objects. It provides insights from mechanical, electrical, hardware and software perspective in the sensor design. The most suitable approach for large object shape sensing using LAFE is also suggested.

Tactile Sensors for Parallel Grippers: Design and Characterization.

Tactile data perception is of paramount importance in today’s robotics applications. This paper describes the latest design of the tactile sensor developed in our laboratory. Both the hardware and firmware concepts are reported in detail in order to allow the research community the sensor reproduction, also according to their needs. The sensor is based on optoelectronic technology and the pad shape can be adapted to various robotics applications. A flat surface, as the one proposed in this paper, can be well exploited if the object sizes are smaller than the pad and/or the shape recognition is needed, while a domed pad can be used to manipulate bigger objects. Compared to the previous version, the novel tactile sensor has a larger sensing area and a more robust electronic, mechanical and software design that yields less noise and higher flexibility. The proposed design exploits standard PCB manufacturing processes and advanced but now commercial 3D printing processes for the realization of all components. A GitHub repository has been prepared with all files needed to allow the reproduction of the sensor for the interested reader. The whole sensor has been tested with a maximum load equal to N, by showing a sensitivity equal to V/N. Moreover, a complete and detailed characterization for the single taxel and the whole pad is reported to show the potentialities of the sensor also in terms of response time, repeatability, hysteresis and signal to noise ratio.

Learning-Based Optoelectronically Innervated Tactile Finger for Rigid-Soft Interactive Grasping

This letter presents a novel design of a soft tactile finger with omni-directional adaptation using multi-channel optical fibers for rigid-soft interactive grasping. Machine learning methods are used to train a model for real-time prediction of force, torque, and contact using the tactile data collected. We further integrated such fingers in a reconfigurable gripper design with three fingers so that the finger arrangement can be actively adjusted in real-time based on the tactile data collected during grasping, achieving the process of rigid-soft interactive grasping. Detailed sensor calibration and experimental results are also included to further validate the proposed design for enhanced grasping robustness. Video: https://www.youtube.com/watch?v=ynCfSA4FQnY .

Lifelong Visual-Tactile Cross-Modal Learning for Robotic Material Perception.

The material attribute of an object's surface is critical to enable robots to perform dexterous manipulations or actively interact with their surrounding objects. Tactile sensing has shown great advantages in capturing material properties of an object's surface. However, the conventional classification method based on tactile information may not be suitable to estimate or infer material properties, particularly during interacting with unfamiliar objects in unstructured environments. Moreover, it is difficult to intuitively obtain material properties from tactile data as the tactile signals about material properties are typically dynamic time sequences. In this article, a visual-tactile cross-modal learning framework is proposed for robotic material perception. In particular, we address visual-tactile cross-modal learning in the lifelong learning setting, which is beneficial to incrementally improve the ability of robotic cross-modal material perception. To this end, we proposed a novel lifelong cross-modal learning model. Experimental results on the three publicly available data sets demonstrate the effectiveness of the proposed method.

Tactile Avatar: Tactile Sensing System Mimicking Human Tactile Cognition.

As a surrogate for human tactile cognition, an artificial tactile perception and cognition system are proposed to produce smooth/soft and rough tactile sensations by its user's tactile feeling; and named this system as “tactile avatar”. A piezoelectric tactile sensor is developed to record dynamically various physical information such as pressure, temperature, hardness, sliding velocity, and surface topography. For artificial tactile cognition, the tactile feeling of humans to various tactile materials ranging from smooth/soft to rough are assessed and found variation among participants. Because tactile responses vary among humans, a deep learning structure is designed to allow personalization through training based on individualized histograms of human tactile cognition and recording physical tactile information. The decision error in each avatar system is less than 2% when 42 materials are used to measure the tactile data with 100 trials for each material under 1.2N of contact force with 4cm s−1 of sliding velocity. As a tactile avatar, the machine categorizes newly experienced materials based on the tactile knowledge obtained from training data. The tactile sensation showed a high correlation with the specific user's tendency. This approach can be applied to electronic devices with tactile emotional exchange capabilities, as well as advanced digital experiences.

Touch Modality Classification Using Recurrent Neural Networks

Recurrent Neural Networks (RNNs) are mainly designed to deal with sequence prediction problems and they show their effectiveness in processing data originally represented as time series. This paper investigates the time series characteristics of RNNs to classify touch modalities represented as spatio temporal 3D tensor data. Different approaches are followed in order to propose efficient RNN models aimed at tactile data classification. The main idea is to capture long-term dependence from data that can be used to deal with long sequences represented by employing Gated Recurrent Unit (GRU) and Long Short-Term Memory (LSTM) architectures. Moreover, a case specific approach to dataset organization of the 3D tensor data is presented. The target is to provide efficient hardware-friendly touch modality classification approaches suitable for embedded applications. To this end, the proposed work achieves effective performance in terms of hardware complexity by reducing the FLOPS by 99.98% and the memory storage by 98.34%, with respect to the state-of-art solutions on the same benchmark dataset. This directly affects the time latency and energy consumption of the embedded hardware. Besides, the implemented models shows a classification accuracy higher than those state-of-art solutions. Results demonstrate that the proposed computing architecture is scalable showing acceptable complexity when the system is scaled up in terms of input matrix size and number of classes to be recognized.

Influence of Surface Tactile Data Quantity on Material Classification in Unstructured Environments

The tactile sensor design and data measurements have been playing an important role in surface material recognition. In inevitable unstructured environments, the performance of the material classification is bottlenecked by multimodal deficiency or data collection intermittency. This article investigates the influence of surface tactile data quantity on material classification in unstructured environments. We extracted tactile data features according to the material-surface-texture model and proposed an approach to determine the scope of the time window of tactile data. We utilized the machine learning classifiers to verify the effectiveness of the hand-designed features and the determined time window. Based on the combination of the Mel frequency cepstrum coefficient and statistics, the majority of the algorithms can classify the tactile data of the upper limit (60 ms in this article) with an accuracy of at least 85%. Exploiting tactile data of the lower limit (50 ms in this article), linear discriminant analysis, quadratic discriminant analysis, and support vector machine can achieve about 93% classification accuracy, and area under curve is about 0.99. The accuracy is not significantly improved with the time window beyond the upper limit, while the performance is degraded and unstable with it beneath the lower limit.

An Efficient Selection-Based kNN Architecture for Smart Embedded Hardware Accelerators

K-Nearest Neighbor (kNN) is an efficient algorithm used in many applications, e.g., text categorization, data mining, and predictive analysis. Despite having a high computational complexity, kNN is a candidate for hardware acceleration since it is a parallelizable algorithm. This paper presents an efficient novel architecture and implementation for a kNN hardware accelerator targeting modern System-on-Chips (SoCs). The architecture adopts a selection-based sorter dedicated for kNN that outperforms traditional sorters in terms of hardware resources, time latency, and energy efficiency. The kNN architecture has been designed using High-Level Synthesis (HLS) and implemented on the Xilinx Zynqberry platform. Compared to similar state-of-the-art implementations, the proposed kNN provides speedups between $1.4\times $ and $875\times $ with 41% to 94% reductions in energy consumption. To further enhance the proposed architecture, algorithmic-level Approximate Computing Techniques (ACTs) have been applied. The proposed approximate kNN implementation accelerates the classification process by $2.3\times $ with an average reduced area size of 56% for a real-time tactile data processing case study. The approximate kNN consumes 69% less energy with an accuracy loss of less than 3% when compared to the proposed Exact kNN.

Transfer of Learning from Vision to Touch: A Hybrid Deep Convolutional Neural Network for Visuo-Tactile 3D Object Recognition.

Transfer of learning or leveraging a pre-trained network and fine-tuning it to perform new tasks has been successfully applied in a variety of machine intelligence fields, including computer vision, natural language processing and audio/speech recognition. Drawing inspiration from neuroscience research that suggests that both visual and tactile stimuli rouse similar neural networks in the human brain, in this work, we explore the idea of transferring learning from vision to touch in the context of 3D object recognition. In particular, deep convolutional neural networks (CNN) pre-trained on visual images are adapted and evaluated for the classification of tactile data sets. To do so, we ran experiments with five different pre-trained CNN architectures and on five different datasets acquired with different technologies of tactile sensors including BathTip, Gelsight, force-sensing resistor (FSR) array, a high-resolution virtual FSR sensor, and tactile sensors on the Barrett robotic hand. The results obtained confirm the transferability of learning from vision to touch to interpret 3D models. Due to its higher resolution, tactile data from optical tactile sensors was demonstrated to achieve higher classification rates based on visual features compared to other technologies relying on pressure measurements. Further analysis of the weight updates in the convolutional layer is performed to measure the similarity between visual and tactile features for each technology of tactile sensing. Comparing the weight updates in different convolutional layers suggests that by updating a few convolutional layers of a pre-trained CNN on visual data, it can be efficiently used to classify tactile data. Accordingly, we propose a hybrid architecture performing both visual and tactile 3D object recognition with a MobileNetV2 backbone. MobileNetV2 is chosen due to its smaller size and thus its capability to be implemented on mobile devices, such that the network can classify both visual and tactile data. An accuracy of 100% for visual and 77.63% for tactile data are achieved by the proposed architecture.

Tactile Perception Technologies and Their Applications in Minimally Invasive Surgery: A Review.

Minimally invasive surgery (MIS) has been the preferred surgery approach owing to its advantages over conventional open surgery. As a major limitation, the lack of tactile perception impairs the ability of surgeons in tissue distinction and maneuvers. Many studies have been reported on industrial robots to perceive various tactile information. However, only force data are widely used to restore part of the surgeon’s sense of touch in MIS. In recent years, inspired by image classification technologies in computer vision, tactile data are represented as images, where a tactile element is treated as an image pixel. Processing raw data or features extracted from tactile images with artificial intelligence (AI) methods, including clustering, support vector machine (SVM), and deep learning, has been proven as effective methods in industrial robotic tactile perception tasks. This holds great promise for utilizing more tactile information in MIS. This review aims to provide potential tactile perception methods for MIS by reviewing literatures on tactile sensing in MIS and literatures on industrial robotic tactile perception technologies, especially AI methods on tactile images.

TSBP: Tangent Space Belief Propagation for Manifold Learning

We present Tangent Space Belief Propagation (TSBP) as a method for graph denoising to improve the robustness of manifold learning algorithms. Dimension reduction by manifold learning relies heavily on the accurate selection of nearest neighbors, which has proven an open problem for sparse and noisy datasets. TSBP uses global nonparametric belief propagation to accurately estimate the tangent spaces of the underlying manifold at each data point. Edges of the neighborhood graph that deviate from the tangent spaces are then removed. The resulting denoised graph can then be embedded into a lower-dimensional space using methods from existing manifold learning algorithms. Artificially generated manifold data, simulated sensor data from a mobile robot, and high dimensional tactile sensory data are used to demonstrate the efficacy of our TSBP method.

Deep Reinforcement Learning for Tactile Robotics: Learning to Type on a Braille Keyboard

Artificial touch would seem well-suited for Reinforcement Learning (RL), since both paradigms rely on interaction with an environment. Here we propose a new environment and set of tasks to encourage development of tactile reinforcement learning: learning to type on a braille keyboard. Four tasks are proposed, progressing in difficulty from arrow to alphabet keys and from discrete to continuous actions. A simulated counterpart is also constructed by sampling tactile data from the physical environment. Using state-of-the-art deep RL algorithms, we show that all of these tasks can be successfully learnt in simulation, and 3 out of 4 tasks can be learned on the real robot. A lack of sample efficiency currently makes the continuous alphabet task impractical on the robot. To the best of our knowledge, this work presents the first demonstration of successfully training deep RL agents in the real world using observations that exclusively consist of tactile images. To aid future research utilising this environment, the code for this project has been released along with designs of the braille keycaps for 3D printing and a guide for recreating the experiments.

Generation of Tactile Data From 3D Vision and Target Robotic Grasps

Tactile perception is a rich source of information for robotic grasping: it allows a robot to identify a grasped object and assess the stability of a grasp, among other things. However, the tactile sensor must come into contact with the target object in order to produce readings. As a result, tactile data can only be attained if a real contact is made. We propose to overcome this restriction by employing a method that models the behaviour of a tactile sensor using 3D vision and grasp information as a stimulus. Our system regresses the quantified tactile response that would be experienced if this grasp were performed on the object. We experiment with 16 items and 4 tactile data modalities to show that our proposal learns this task with low error.

Investigations on the ship-ice impact: Part 1. Experimental methodologies

In the structural design of the Polar Class ships, glancing impact with ice has been considered as the governing load scenario for dimensioning the bow structure. At present, non-linear transient dynamic analysis has been reckoned as a fundamental requirement in the assessment of the structural strength of ships with higher ice classes. Under such requirement, understanding of the dynamic characteristics of the ship-ice impact load appears significant importance. The present study aims to investigate the spatial and temporal variations of the impact loads by simulating the glancing impact events between a polar research vessel and giant ice floes in ice tank. The ship-ice impact loads were recorded by a grid-based tactile sensor attached on the bow area of the model ship. To achieve a reasonable simulation on the design scenario described by the IACS Unified Requirements for Polar Class Ships (UR I), a series of methodological calibration tests were preliminarily carried out to determine the key parameters that should be carefully monitored and controlled, accompanying with thorough discussions on the ship-ice impact process. This paper provides detailed information on the preliminary methodological calibrations and the tactile data processing techniques, including the identification of the ship-ice contact area, the depiction of the ice loading trail and the outline of the spatial distributions of local ice pressures. A companion paper provides detailed analyses and discussions regarding the spatial and temporal variation of the ice impact loads from the formal tests based on the proposed testing procedures obtained by the methodological calibrations.