Tactile Signals Research Articles

Some, but not all, cochlear implant users prefer music stimuli with congruent haptic stimulation.

Cochlear implant (CI) users often report being unsatisfied by music listening through their hearing device. Vibrotactile stimulation could help alleviate those challenges. Previous research has shown that musical stimuli was given higher preference ratings by normal-hearing listeners when concurrent vibrotactile stimulation was congruent in intensity and timing with the corresponding auditory signal compared to incongruent. However, it is not known whether this is also the case for CI users. Therefore, in this experiment, we presented 18 CI users and 24 normal-hearing listeners with five melodies and five different audio-to-tactile maps. Each map varied the congruence between the audio and tactile signals related to intensity, fundamental frequency, and timing. Participants were asked to rate the maps from zero to 100, based on preference. It was shown that almost all normal-hearing listeners, as well as a subset of the CI users, preferred tactile stimulation, which was congruent with the audio in intensity and timing. However, many CI users had no difference in preference between timing aligned and timing unaligned stimuli. The results provide evidence that vibrotactile music enjoyment enhancement could be a solution for some CI users; however, more research is needed to understand which CI users can benefit from it most.

Tactile texture recognition of multi-modal bionic finger based on multi-modal CBAM-CNN interpretable method

To solve the problem of lacking surface information while using a single-modal sensor to scan fine textures, a multi-modal bionic finger tactile sensor is used to perceive the information contained in textures, and a multi-modal tactile texture recognition method is proposed to address the issue of poor interpretability for tactile texture recognition models. First, the experimental platform of multi-modal bionic finger tactile texture information acquisition system is built, which can simultaneously collect 14 channels of tactile electrode signals, one channel of pressure signals and one channel of temperature signals. Second, the multi-modal tactile signals can be fused at the data level after signal processing, and then the wavelet transform method is carried out on the basis of data fusion. Third, a multi-modal CBAM-CNN network (Convolutional Block Attention Module, Convolutional Neural Network) is established to extract the multi-modal tactile features after the wavelet transform and the multi-modal tactile features are fused at feature level. To verify the performance of multi-modal CBAM-CNN model, 27(3*9) kinds of millimeter-level fine textures of 3 datasets are tested and the recognition accuracy of each dataset is 98.47%, 97.63%, 99.16%. And it can also be seen that the 14-channel modal data contributes the most to the performance of multi-modal CBAM-CNN model for various textures. The results of ablation experiments prove that the identification accuracy of the fused multi-modal CBAM-CNN is higher than the single-modal CBAM-CNN no matter in time domain or time–frequency domain. Moreover, the accuracy of the weighted fusion method based on correlation function is higher than horizontal fusion method in 14 channels of tactile electrode signals modal. Finally, to interpret the classification decision of the multi-modal CBAM-CNN network in each modal intuitively, the improved multi-modal CAM (Class Activation Map) algorithm is adopted to visualize the interest regions of the samples in detail, and the activated areas in the map make the identification result of CBAM-CNN in each modal more transparent and interpretable.

Role of arm reaching movement kinematics in friction perception at initial contact with smooth surfaces.

When manipulating objects, humans begin adjusting their grip force to friction within 100ms of contact. During motor adaptation, subjects become aware of the slipperiness of touched surfaces. Previously, we have demonstrated that humans cannot perceive frictional differences when surfaces are brought in contact with an immobilised finger, but can do so when there is submillimeter lateral displacement or subjects actively make the contact movement. Similarly, in, we investigated how humans perceive friction in the absence of intentional exploratory sliding or rubbing movements, to mimic object manipulation interactions. We used a two-alternative forced-choice paradigm in which subjects had to reach and touch one surface followed by another, and then indicate which felt more slippery. Subjects correctly identified the more slippery surface in 87±8% of cases (mean±SD; n=12). Biomechanical analysis of finger pad skin displacement patterns revealed the presence of tiny (<1mm) localised slips, known to be sufficient to perceive frictional differences. We tested whether these skin movements arise as a result of natural hand reaching kinematics. The task was repeated with the introduction of a hand support, eliminating the hand reaching movement and minimising fingertip movement deviations from a straight path. As a result, our subjects' performance significantly declined (66±12% correct, mean±SD; n=12), suggesting that unrestricted reaching movement kinematics and factors such as physiological tremor, play a crucial role in enhancing or enabling friction perception upon initial contact. KEY POINTS: More slippery objects require a stronger grip to prevent them from slipping out of hands. Grip force adjustments to friction driven by tactile sensory signals are largely automatic and do not necessitate cognitive involvement; nevertheless, some associated awareness of grip surface slipperiness under such sensory conditions is present and helps to select a safe and appropriate movement plan. When gripping an object, tactile receptors provide frictional information without intentional rubbing or sliding fingers over the surface. However, we have discovered that submillimeter range lateral displacement might be required to enhance or enable friction sensing. The present study provides evidence that such small lateral movements causing localised partial slips arise and are an inherent part of natural reaching movement kinematics.

Jointly Modeling Spatio-Temporal Features of Tactile Signals for Action Classification

Tactile signals collected by wearable electronics are essential in modeling and understanding human behavior. One of the main applications of tactile signals is action classification, especially in healthcare and robotics. However, existing tactile classification methods fail to capture the spatial and temporal features of tactile signals simultaneously, which results in sub-optimal performances. In this paper, we design Spatio-Temporal Aware tactility Transformer (STAT) to utilize continuous tactile signals for action classification. We propose spatial and temporal embeddings along with a new temporal pretraining task in our model, which aims to enhance the transformer in modeling the spatio-temporal features of tactile signals. Specially, the designed temporal pretraining task is to differentiate the time order of tubelet inputs to model the temporal properties explicitly. Experimental results on a public action classification dataset demonstrate that our model outperforms state-of-the-art methods in all metrics.

Industrially Scalable Textile Sensing Interfaces for Extended Artificial Tactile and Human Motion Monitoring without Compromising Comfort.

Smart wearables with the capability for continuous monitoring, perceiving, and understanding human tactile and motion signals, while ensuring comfort, are highly sought after for intelligent healthcare and smart life systems. However, concurrently achieving high-performance tactile sensing, long-lasting wearing comfort, and industrialized fabrication by a low-cost strategy remains a great challenge. This is primarily due to critical research gaps in novel textile structure design for seamless integration with sensing elements. Here, an all-in-one biaxial insertion knit architecture is reported to topologically integrate sensing units within double-knit loops for the fabrication of a large-scale tactile sensing textile by using low-cost industrial manufacturing routes. High sensitivity, stability, and low hysteresis of arrayed sensing units are achieved through engineering of fractal structures of hierarchically patterned piezoresistive yarns via blistering and twisting processing. The as-prepared tactile sensing textiles show desirable sensing performance and robust mechanical property, while ensuring excellent conformability, tailorability, breathability (288 mm s-1), and moisture permeability (3591 g m-2 per day) for minimizing the effect on wearing comfort. The multifunctional applications of tactile sensing textiles are demonstrated in continuously monitoring human motions, tactile interactions with the environment, and recognizing biometric gait. Moreover, we also demonstrate that machine learning-assisted sensing textiles can accurately predict body postures, which holds great promise in advancing the development of personalized healthcare robotics, prosthetics, and intelligent interaction devices.

Ceramic nanoparticle based flexible hydrovoltaic devices for tactile and respiratory signal detection

Devices that utilize water evaporation or moisture to generate electricity have received significant research attention in the field of energy conversion and harvesting. Herein, a device capable of generating electricity through the process of water evaporation has been created, utilizing medical nonwoven fabric as a porous substrate with simple dip-coating and spray-coating techniques. The device constructs a multitude of nanopores on the substrate by employing a blend of Al2O3 nanoparticles of varying dimensions. The electrical output of the device was stable at 0.55 V and 6 μA from a small water supply, as confirmed by the test. The device can easily recognize the click of a finger through the release of minuscule amounts of water during the contact. Furthermore, the device has the ability to recognize various breathing patterns, such as normal breathing, rapid breathing, and wheezing that may induced by diseases. This work developed a flexible device for power generation by water evaporation process and provides a referable idea for human related humidity sensing.

ASICs mediate fast excitatory synaptic transmission for tactile discrimination

Tactile discrimination, the ability to differentiate objects' physical properties such as texture, shape, and edges, is essential for environmental exploration, social interaction, and early childhood development. This ability heavily relies on Merkel cell-neurite complexes (MNCs), the tactile end-organs enriched in the fingertips of humans and the whisker hair follicles of non-primate mammals. Although recent studies have advanced our knowledge on mechanical transduction in MNCs, it remains unknown how tactile signals are encoded at MNCs. Here, using rodent whisker hair follicles, we show that tactile signals are encoded at MNCs as fast excitatory synaptic transmission. This synaptic transmission is mediated by acid-sensing ion channels (ASICs) located on the neurites of MNCs, with protons as the principal transmitters. Pharmacological inhibition or genetic deletion of ASICs diminishes the tactile encoding at MNCs and impairs tactile discrimination in animals. Together, ASICs are required for tactile encoding at MNCs to enable tactile discrimination in mammals.

Machine learning-coupled vertical graphene triboelectric pressure sensors array as artificial tactile receptor for finger action recognition

Wearable pressure sensors have been emerging for recording human biomechanical information during movement and action behavior. When combined with advanced signals processing and analysis methods, these sensors may offer the opportunities of recognizing patterns in specific human actions. In this study, we proposed a machine learning-coupled vertical graphene triboelectric pressure sensors array (ML-vGTEPS array), which could serve as artificial finger tactile receptors for sensing pressure patterns generated by specific finger actions. By utilizing the vertical graphene (vG) with excellent electrical properties and unique nanostructure as the key friction layer, the triboelectric sensor exhibited high sensitivity and broad sensing range. Ten sets of triboelectric sensors are integrated and designed as wearable devices on fingers, which could transmit multi-channel tactile signals under different finger actions with minimal crosstalk. Taking the tracking technical actions of playing table tennis as an example, 16 types of finger tactile signals generated during specific table tennis technical actions were obtained using a self-designed acquisition hardware. Action recognition with high accuracy (98.1%) was achieved by a fully connected neural network (FCNN) deep learning model, serving as validation for the ML-vGTEPS Array in action monitoring and recognition. The exemplary recognition results demonstrated the potential of the ML-vGTEPS Array as a high-performance technique for human-machine interface, intelligent athletic training, telemedicine, and applications in virtual reality/augmented reality.

Active self-touch restores bodily proprioceptive spatial awareness following disruption by 'rubber hand illusion'.

Bodily self-awareness relies on a constant integration of visual, tactile, proprioceptive, and motor signals. In the 'rubber hand illusion' (RHI), conflicting visuo-tactile stimuli lead to changes in self-awareness. It remains unclear whether other, somatic signals could compensate for the alterations in self-awareness caused by visual information about the body. Here, we used the RHI in combination with robot-mediated self-touch to systematically investigate the role of tactile, proprioceptive and motor signals in maintaining and restoring bodily self-awareness. Participants moved the handle of a leader robot with their right hand and simultaneously received corresponding tactile feedback on their left hand from a follower robot. This self-touch stimulation was performed either before or after the induction of a classical RHI. Across three experiments, active self-touch delivered after-but not before-the RHI, significantly reduced the proprioceptive drift caused by RHI, supporting a restorative role of active self-touch on bodily self-awareness. The effect was not present during involuntary self-touch. Unimodal control conditions confirmed that both tactile and motor components of self-touch were necessary to restore bodily self-awareness. We hypothesize that active self-touch transiently boosts the precision of proprioceptive representation of the touched body part, thus counteracting the visual capture effects that underlie the RHI.

Enhanced braille recognition based on piezoresistive and piezoelectric dual-mode tactile sensors

Human Tactile perception can be divided into pressure perception and slip perception according to the mode of contact. At present, however, tactile sensors can not have the same good performance as human pressure and slip perception. In this paper, a piezoresistive and piezoelectric dual-mode tactile sensing scheme is proposed based on human finger tactile skin structure. PVDF polymer is used as the sensitive material in piezoelectric layer, which has good dynamic response and high sensitivity. Graphene and carbon black are used as the sensitive material in piezoresistive layer, the array process is simple and high resolution can be achieved. The fast response of piezoelectric signal can be used to segment the piezoresistive signal and improve the recognition accuracy when the tactile sensor is used for slip sensing. At the same time, the piezoelectric layer can also reflect the velocity information and direction information. In recognition method, according to the feature that the continuous tactile signal has both temporal and spatial features in the process of sliding, CNN-LSTM is proposed in this paper. The tactile sensors is integrated into the robot arm to detect 20 kinds of braille. At the same speed and the same force, the recognition rate of bimodal braille recognition is 90.58%. In addition, the model can be well recognized at different speeds and sliding directions, and the recognition rate is 84.2%. Finally, the tactile sensor is wore on the index finger of human hand, the average recognition rate of touch braille is 77.5%, which shows that the algorithm has good robustness, and the speed of recognition is twice as fast as that of the normal blind. In the future, it can be applied to blind reading, surface detection, texture identification and other fields.

Establishing Natural Tactile Mappings: Mapping Tactile Parameters to Continuous Data Concepts.

There has been limited work developing natural mappings between tactile signals and common data concepts in data rich domains. If these mappings can be established, tactile displays can become more intuitive and readily adopted. The present study aims to identify general natural mappings between perceptual dimensions of vibration and continuous data concepts. Twenty-one participants were tasked to map four different tactile parameters to four different data concepts-pressure, concentration, size, and speed. We found that an increase in intensity was good at conveying increases for all data concepts. We also found that speed, pressure, concentration, and size all have at least one strong natural mapping.

Perceiving Synchrony: Determining Thermal-tactile Simultaneity Windows.

In cutaneous displays in which both tactile and thermal signals are presented, it is important to understand the temporal requirements associated with presenting these signals so that they are perceptually synchronous. Such synchrony is important to provide realistic touch experiences in applications involving object recognition and social touch interactions. In the present experiment the temporal window within which tactile and warm thermal stimuli are perceived to occur at the same time was determined. A Simultaneity Judgment Task was used in which pairs of tactile and thermal stimuli were presented on the hand at varying stimulus onset asynchronies, and participants determined whether the stimuli were simultaneous or not. The results indicated that the average simultaneity window width was 1041 ms. The average point of subjective simultaneity (PSS) was -569 ms, indicating that participants perceived simultaneity best when the warm thermal stimulus preceded the tactile stimulus by 569 ms. These findings indicate that thermal and tactile stimuli do not need to be displayed simultaneously for the two stimuli to be perceived as being synchronous and therefore the timing of such stimuli can be adjusted to maximize the likelihood that they will both be perceived.

Beyond End-Effector: Utilizing High-Resolution Tactile Signals for Physical Human–Robot Interaction

Beyond End-Effector: Utilizing High-Resolution Tactile Signals for Physical Human–Robot Interaction

Are blind individuals immune to bodily illusions? Somatic rubber hand illusion in the blind revisited

Multisensory awareness of one’s own body relies on the integration of signals from various sensory modalities such as vision, touch, and proprioception. But how do blind individuals perceive their bodies without visual cues, and does the brain of a blind person integrate bodily senses differently from a sighted person? To address this question, we aimed to replicate the only two previous studies on this topic, which claimed that blind individuals do not experience the somatic rubber hand illusion, a bodily illusion triggered by the integration of correlated tactile and proprioceptive signals from the two hands. We used a larger sample size than the previous studies and added Bayesian analyses to examine statistical evidence in favor of the lack of an illusion effect. Moreover, we employed tests to investigate whether enhanced tactile acuity and cardiac interoceptive accuracy in blind individuals could also explain the weaker illusion. We tested 36 blind individuals and 36 age- and sex-matched sighted volunteers. The results show that blind individuals do not experience the somatic rubber hand illusion based on questionnaire ratings and behavioral measures that assessed changes in hand position sense toward the location of the rubber hand. This conclusion is supported by Bayesian evidence in favor of the null hypothesis. The findings confirm that blind individuals do not experience the somatic rubber hand illusion, indicating that lack of visual experience leads to permanent changes in multisensory bodily perception. In summary, our study suggests that changes in multisensory integration of tactile and proprioceptive signals may explain why blind individuals are “immune” to the nonvisual version of the rubber hand illusion.

CAvatar

Human mesh reconstruction is essential for various applications, including virtual reality, motion capture, sports performance analysis, and healthcare monitoring. In healthcare contexts such as nursing homes, it is crucial to employ plausible and non-invasive methods for human mesh reconstruction that preserve privacy and dignity. Traditional vision-based techniques encounter challenges related to occlusion, viewpoint limitations, lighting conditions, and privacy concerns. In this research, we present CAvatar, a real-time human mesh reconstruction approach that innovatively utilizes pressure maps recorded by a tactile carpet as input. This advanced, non-intrusive technology obviates the need for cameras during usage, thereby safeguarding privacy. Our approach addresses several challenges, such as the limited spatial resolution of tactile sensors, extracting meaningful information from noisy pressure maps, and accommodating user variations and multiple users. We have developed an attention-based deep learning network, complemented by a discriminator network, to predict 3D human pose and shape from 2D pressure maps with notable accuracy. Our model demonstrates promising results, with a mean per joint position error (MPJPE) of 5.89 cm and a per vertex error (PVE) of 6.88 cm. To the best of our knowledge, we are the first to generate 3D mesh of human activities solely using tactile carpet signals, offering a novel approach that addresses privacy concerns and surpasses the limitations of existing vision-based and wearable solutions. The demonstration of CAvatar is shown at https://youtu.be/ZpO3LEsgV7Y.

Human-Inspired Tactile Perception System for Real-Time and Multimodal Detection of Tactile Stimuli.

A human can intuitively perceive and comprehend complicated tactile information because the cutaneous receptors distributed in the fingertip skin receive different tactile stimuli simultaneously and the tactile signals are immediately transmitted to the brain. Although many research groups have attempted to mimic the structure and function of human skin, it remains a challenge to implement human-like tactile perception process inside one system. In this study, we developed a real-time and multimodal tactile system that mimics the function of cutaneous receptors and the transduction of tactile stimuli from receptors to the brain, by using multiple sensors, a signal processing and transmission circuit module, and a signal analysis module. The proposed system is capable of simultaneously acquiring four types of decoupled tactile information with a compact system, thereby enabling differentiation between various tactile stimuli, texture characteristics, and consecutive complex motions. This skin-like three-dimensional integrated design provides further opportunities in multimodal tactile sensing systems.

Magnetically Induced Grid Structure for Enhancing the Performance of a Dual-Mode Flexible Sensor with Tactile/Touchless Perception.

As an advanced sensing technology, dual-mode flexible sensing, integrating both tactile and touchless perception, propels numerous intelligent devices toward a more practical and efficient direction. The ability to incorporate multiple sensing modes and accurately distinguish them in real time has become crucial for technological advancements. Here, we proposed a dual-mode sensing system (B-MIGS) consisting of a dual-layer sensing device with a magnetically induced grid structure and a testing device. The system was capable of utilizing mechanical pressure to perceive tactile stimulation and magnetic sensing to simultaneously transduce touchless stimulation simultaneously. By leveraging the triboelectric effect, the decoupling of tactile and touchless signals in the presence of unknown signal sources was achieved. Additionally, the sensing characteristics of the B-MIGS were optimized by varying the curing magnetic induction intensity and magnetic particle concentration. The influence of the temperature and humidity on the sensing signals was also discussed. Finally, the practical value of the B-MIGS as a dual-mode monitoring system was demonstrated on soft petals and sensor arrays, along with exploration of its potential application in underwater environments.

Associative memory neurons of encoding multi-modal signals are recruited by neuroligin-3-mediated new synapse formation

The joint storage and reciprocal retrieval of learnt associated signals are presumably encoded by associative memory cells. In the accumulation and enrichment of memory contents in lifespan, a signal often becomes a core signal associatively shared for other signals. One specific group of associative memory neurons that encode this core signal likely interconnects multiple groups of associative memory neurons that encode these other signals for their joint storage and reciprocal retrieval. We have examined this hypothesis in a mouse model of associative learning by pairing the whisker tactile signal sequentially with the olfactory signal, the gustatory signal, and the tail-heating signal. Mice experienced this associative learning show the whisker fluctuation induced by olfactory, gustatory, and tail-heating signals, or the other way around, that is, memories to multi-modal associated signals featured by their reciprocal retrievals. Barrel cortical neurons in these mice become able to encode olfactory, gustatory, and tail-heating signals alongside the whisker signal. Barrel cortical neurons interconnect piriform, S1-Tr, and gustatory cortical neurons. With the barrel cortex as the hub, the indirect activation occurs among piriform, gustatory, and S1-Tr cortices for the second-order associative memory. These associative memory neurons recruited to encode multi-modal signals in the barrel cortex for associative memory are downregulated by neuroligin-3 knockdown. Thus, associative memory neurons can be recruited as the core cellular substrate to memorize multiple associated signals for the first-order and the second-order of associative memories by neuroligin-3-mediated synapse formation, which constitutes neuronal substrates of cognitive activities in the field of memoriology.

Associative memory neurons of encoding multi-modal signals are recruited by neuroligin-3-mediated new synapse formation.

The joint storage and reciprocal retrieval of learnt associated signals are presumably encoded by associative memory cells. In the accumulation and enrichment of memory contents in lifespan, a signal often becomes a core signal associatively shared for other signals. One specific group of associative memory neurons that encode this core signal likely interconnects multiple groups of associative memory neurons that encode these other signals for their joint storage and reciprocal retrieval. We have examined this hypothesis in a mouse model of associative learning by pairing the whisker tactile signal sequentially with the olfactory signal, the gustatory signal, and the tail-heating signal. Mice experienced this associative learning show the whisker fluctuation induced by olfactory, gustatory, and tail-heating signals, or the other way around, that is, memories to multi-modal associated signals featured by their reciprocal retrievals. Barrel cortical neurons in these mice become able to encode olfactory, gustatory, and tail-heating signals alongside the whisker signal. Barrel cortical neurons interconnect piriform, S1-Tr, and gustatory cortical neurons. With the barrel cortex as the hub, the indirect activation occurs among piriform, gustatory, and S1-Tr cortices for the second-order associative memory. These associative memory neurons recruited to encode multi-modal signals in the barrel cortex for associative memory are downregulated by neuroligin-3 knockdown. Thus, associative memory neurons can be recruited as the core cellular substrate to memorize multiple associated signals for the first-order and the second-order of associative memories by neuroligin-3-mediated synapse formation, which constitutes neuronal substrates of cognitive activities in the field of memoriology.

Callosal inputs generate side-invariant receptive fields in the barrel cortex.

Barrel cortex integrates contra- and ipsilateral whiskers' inputs. While contralateral inputs depend on the thalamocortical innervation, ipsilateral ones are thought to rely on callosal axons. These are more abundant in the barrel cortex region bordering with S2 and containing the row A-whiskers representation, the row lying nearest to the facial midline. Here, we ask what role this callosal axonal arrangement plays in ipsilateral tactile signaling. We found that novel object exploration with ipsilateral whiskers confines c-Fos expression within the highly callosal subregion. Targeting this area with in vivo patch-clamp recordings revealed neurons with uniquely strong ipsilateral responses dependent on the corpus callosum, as assessed by tetrodotoxin silencing and by optogenetic activation of the contralateral hemisphere. Still, in this area, stimulation of contra- or ipsilateral row A-whiskers evoked an indistinguishable response in some neurons, mostly located in layers 5/6, indicating their involvement in the midline representation of the whiskers' sensory space.