Tactile Flow Research Articles

Visuo-tactile heading perception.

Self-motion through an environment induces various sensory signals, i.e., visual, vestibular, auditory, or tactile. Numerous studies have investigated the role of visual and vestibular stimulation for the perception of self-motion direction (heading). Here, we investigated the rarely considered interaction of visual and tactile stimuli in heading perception. Participants were presented optic flow simulating forward self-motion across a horizontal ground plane (visual), airflow toward the participants' forehead (tactile), or both. In separate blocks of trials, participants indicated perceived heading from unimodal visual or tactile or bimodal sensory signals. In bimodal trials, presented headings were either spatially congruent or incongruent with a maximum offset between visual and tactile heading of 30°. To investigate the reference frame in which visuo-tactile heading is encoded, we varied head and eye orientation during presentation of the stimuli. Visual and tactile stimuli were designed to achieve comparable precision of heading reports between modalities. Nevertheless, in bimodal trials heading perception was dominated by the visual stimulus. A change of head orientation had no significant effect on perceived heading, whereas, surprisingly, a change in eye orientation affected tactile heading perception. Overall, we conclude that tactile flow is more important to heading perception than previously thought.NEW & NOTEWORTHY We investigated heading perception from visual-only (optic flow), tactile-only (tactile flow), or bimodal self-motion stimuli in different conditions varying in head and eye position. Overall, heading perception was body or world centered and non-Bayes optimal and revealed a centripetal bias. Although being visually dominated, tactile flow revealed a significant influence during bimodal heading perception.

3D Contact Point Cloud Reconstruction From Vision-Based Tactile Flow

With the growing interest in vision-based tactile sensors, various types of sensors that utilize digital imaging are being developed. Among them, a group of sensors captures the tactile flow resulted from the contact deformation using the optical flow algorithm from computer vision and achieves full resolution deformation tracking on the tactile surface. In this work, a novel 3D contact reconstruction algorithm is proposed and evaluated. It exploits the contact geometry and projection relationship in the tactile flow, which are versatile for vision-based tactile sensors, unique for tactile perception but not inherited from computer vision. The resulted 3D contact point cloud representation is consistent with the tactile flow constraint, scale estimation, and contact edge estimation. It can be directly manipulated in downstream applications such as contact force estimation and contact pose estimation. Experiments and examples are provided that indicate the potential for the proposed tactile processing algorithm to connect tactile perception to tactile enabled robotic manipulation tasks.

Influence of Tactile Flow on Visual Heading Perception.

The integration of information from different sensory modalities is crucial for successful navigation through an environment. Among others, self-motion induces distinct optic flow patterns on the retina, vestibular signals and tactile flow, which contribute to determine traveled distance (path integration) or movement direction (heading). While the processing of combined visual-vestibular information is subject to a growing body of literature, the processing of visuo-tactile signals in the context of self-motion has received comparatively little attention. Here, we investigated whether visual heading perception is influenced by behaviorally irrelevant tactile flow. In the visual modality, we simulated an observer's self-motion across a horizontal ground plane (optic flow). Tactile self-motion stimuli were delivered by air flow from head-mounted nozzles (tactile flow). In blocks of trials, we presented only visual or tactile stimuli and subjects had to report their perceived heading. In another block of trials, tactile and visual stimuli were presented simultaneously, with the tactile flow within ±40° of the visual heading (bimodal condition). Here, importantly, participants had to report their perceived visual heading. Perceived self-motion direction in all conditions revealed a centripetal bias, i.e., heading directions were perceived as compressed toward straight ahead. In the bimodal condition, we found a small but systematic influence of task-irrelevant tactile flow on visually perceived headings as function of their directional offset. We conclude that tactile flow is more tightly linked to self-motion perception than previously thought.

On the Role of Lateral Force in Texture-Induced Motion Bias During Reaching Tasks.

In the previous work, we reported that tactile information (tactile slip) during finger sliding and reaching actions over a textured surface contributes to the control of the hand movement. More specifically, we observed a significant bias in the motion trajectories, which was explained by the tactile estimate accounted by the tactile flow model-i.e. a perceived motion direction always perpendicular to the ridge orientation, and its integration with the muscular-skeletal proprioceptive cues. However, to which extent this observed behavior also depends on the reaction force exerted by the surface ridges on the finger pad during the dynamic interaction still represents a largely unexplored research question. If not properly addressed, this point could rise the alternative explanation that the systematic bias is determined by the insufficient compensation of the reaction force by participants. In this article, we investigate the role of the lateral component of the reaction force on the surface plane (lateral force) in texture-related motion bias. We asked participants to slide their finger straight on a lubricated ridged plate towards a target goal displayed in a virtual reality environment. They exerted two different levels of normal force, which produced two different levels of lateral force during the finger interaction with the ridges. The effect of ridge orientation was found to be larger for the high compared to the low force level. However, also in the latter case, we still observed the same biased trajectories reported in our previous work, despite the negligible value of the lateral force. This supports our hypothesis that the motor bias arises from the integration of the tactile motion estimate, biased by the texture, and the other proprioceptive cues.

Layer-specific integration of locomotion and sensory information in mouse barrel cortex

During navigation, rodents continually sample the environment with their whiskers. How locomotion modulates neuronal activity in somatosensory cortex, and how it is integrated with whisker-touch remains unclear. Here, we compared neuronal activity in layer 2/3 (L2/3) and L5 of barrel cortex using calcium imaging in mice running in a tactile virtual reality. Both layers increase their activity during running and concomitant whisking, in the absence of touch. Fewer neurons are modulated by whisking alone. Whereas L5 neurons respond transiently to wall-touch during running, L2/3 neurons show sustained activity. Consistently, neurons encoding running-with-touch are more abundant in L2/3 and they encode the run-speed better during touch. Few neurons across layers were also sensitive to abrupt perturbations of tactile flow during running. In summary, locomotion significantly enhances barrel cortex activity across layers with L5 neurons mainly reporting changes in touch conditions and L2/3 neurons continually integrating tactile stimuli with running.

End-to-End ConvNet for Tactile Recognition Using Residual Orthogonal Tiling and Pyramid Convolution Ensemble

Tactile recognition enables robots identify target objects or environments from tactile sensory readings. The recent advancement of deep learning and biological tactile sensing inspire us proposing an end-to-end architecture ROTConvPCE-mv that performs tactile recognition using residual orthogonal tiling and pyramid convolution ensemble. Our approach uses stacks of raw frames and tactile flow as dual input, and incorporates the strength of multi-layer OTConvs (orthogonal tiling convolutions) organized in a residual learning paradigm. We empirically demonstrate that OTConvs have adjustable invariance capability to different input transformations such as translation, rotation, and scaling. To effectively capture multi-scale global context, a pyramid convolution structure is attached to the concatenated output of two residual OTConv pathways. The extensive experimental evaluations show that ROTConvPCE-mv outperforms several state-of-the-art methods with a large margin regarding recognition accuracy, robustness, and fault-tolerance. Practical suggestions and hints are summarized throughout this paper to facilitate the effective recognition using tactile sensory data.

Tactile Flow Overrides Other Cues To Self Motion

Vestibular-somatosensory interactions are pervasive in the brain but it remains unclear why. Here we explore the contribution of tactile flow to processing self-motion. We assessed two aspects of self-motion: timing and speed. Participants sat on an oscillating swing and either kept their hands on their laps or rested them lightly on an earth-stationary surface. They viewed a grating oscillating at the same frequency as their motion and judged its phase or, in a separate experiment, its speed relative to their perceived motion. Participants required the phase to precede body movement (with or without tactile flow) or tactile flow by ~5° (44 ms) to appear earth-stationary. Speed judgments were 4–10% faster when motion was from tactile flow, either alone or with body motion, compared to body motion alone (where speed judgments were accurate). By comparing response variances we conclude that phase and speed judgments do not reflect optimal integration of tactile flow with other cues to body motion: instead tactile flow dominates perceived self-motion – acting as an emergency override. This may explain why even minimal tactile cues are so helpful in promoting stability and suggests that providing artificial tactile cues might be a powerful aid to perceiving self-motion.

Tactile Apparent Motion on the Torso Modulates Perceived Forward Self-Motion Velocity.

The present study investigated whether a tactile flow created by a matrix of vibrators in a seat pan simultaneously presented with an optical flow in peripheral vision enhances the perceived forward velocity of self-motion. A brief tactile motion stimulus consisted of four successive rows of vibration, and the interstimulus onset between the tactile rows was varied to change the velocity of the tactile motion. The results show that the forward velocity of self-motion is significantly overestimated for rapid tactile flows and underestimated for slow ones, compared with optical flow alone or non-motion vibrotactile stimulation conditions. In addition, the effect with a temporal tactile rhythm without changing the stimulus location was smaller than that with spatiotemporal tactile motion, with the interstimulus onset interval to elicit a clear sensation of tactile apparent motion. These findings suggest that spatiotemporal tactile motion is effective in inducing a change in the perceived forward velocity of self-motion.

A Finite element model of tactile flow for softness perception.

Touch is an extremely dynamic sense. To take into account this aspect, it has been hypothesized that there are mechanisms in the brain that specialize in processing dynamic tactile stimuli, in a way not too dissimilar from what happens for optical flow in dynamic vision. The concept of tactile flow, related to the rate of expansion of isostrain volumes in the human fingerpad, was used to explain some perceptual illusions as well as mechanisms of human softness perception. In this paper we describe a computational model of tactile flow, and apply it to a finite element model of interaction between deformable bodies. The shape and material properties of the bodies are modeled from those of a human fingertip interacting with specimens with different softness properties. Results show that the rate of expansion of isostrain volumes can be used to discriminate different materials in terms of their softness characteristics.

Identification of Vibrotactile Patterns Encoding Obstacle Distance Information.

Delivering distance information of nearby obstacles from sensors embedded in a white cane-in addition to the intrinsic mechanical feedback from the cane-can aid the visually impaired in ambulating independently. Haptics is a common modality for conveying such information to cane users, typically in the form of vibrotactile signals. In this context, we investigated the effect of tactile rendering methods, tactile feedback configurations and directions of tactile flow on the identification of obstacle distance. Three tactile rendering methods with temporal variation only, spatio-temporal variation and spatial/temporal/intensity variation were investigated for two vibration feedback configurations. Results showed a significant interaction between tactile rendering method and feedback configuration. Spatio-temporal variation generally resulted in high correct identification rates for both feedback configurations. In the case of the four-finger vibration, tactile rendering with spatial/temporal/intensity variation also resulted in high distance identification rate. Further, participants expressed their preference for the four-finger vibration over the single-finger vibration in a survey. Both preferred rendering methods with spatio-temporal variation and spatial/temporal/intensity variation for the four-finger vibration could convey obstacle distance information with low workload. Overall, the presented findings provide valuable insights and guidance for the design of haptic displays for electronic travel aids for the visually impaired.

Sensory prediction on a whiskered robot: a tactile analogy to “optical flow”

When an animal moves an array of sensors (e.g., the hand, the eye) through the environment, spatial and temporal gradients of sensory data are related by the velocity of the moving sensory array. In vision, the relationship between spatial and temporal brightness gradients is quantified in the “optical flow” equation. In the present work, we suggest an analog to optical flow for the rodent vibrissal (whisker) array, in which the perceptual intensity that “flows” over the array is bending moment. Changes in bending moment are directly related to radial object distance, defined as the distance between the base of a whisker and the point of contact with the object. Using both simulations and a 1×5 array (row) of artificial whiskers, we demonstrate that local object curvature can be estimated based on differences in radial distance across the array. We then develop two algorithms, both based on tactile flow, to predict the future contact points that will be obtained as the whisker array translates along the object. The translation of the robotic whisker array represents the rat's head velocity. The first algorithm uses a calculation of the local object slope, while the second uses a calculation of the local object curvature. Both algorithms successfully predict future contact points for simple surfaces. The algorithm based on curvature was found to more accurately predict future contact points as surfaces became more irregular. We quantify the inter-related effects of whisker spacing and the object's spatial frequencies, and examine the issues that arise in the presence of real-world noise, friction, and slip.

Multisensory space: from eye‐movements to self‐motion

We perceive the world around us as stable. This is remarkable given that our body parts as well as we ourselves are constantly in motion. Humans and other primates move their eyes more often than their hearts beat. Such eye movements lead to coherent motion of the images of the outside world across the retina. Furthermore, during everyday life, we constantly approach targets, avoid obstacles or otherwise move in space. These movements induce motion across different sensory receptor epithels: optical flow across the retina, tactile flow across the body surface and even auditory flow as detected from the two ears. It is generally assumed that motion signals as induced by one's own movement have to be identified and differentiated from the real motion in the outside world. In a number of experimental studies we and others have functionally characterized the primate posterior parietal cortex (PPC) and its role in multisensory encoding of spatial and motion information. Extracellular recordings in the macaque monkey showed that during steady fixation the visual, auditory and tactile spatial representations in the ventral intraparietal area (VIP) are congruent. This finding was of major importance given that a functional MRI (fMRI) study determined the functional equivalent of macaque area VIP in humans. Further recordings in other areas of the dorsal stream of the visual cortical system of the macaque pointed towards the neural basis of perceptual phenomena (heading detection during eye movements, saccadic suppression, mislocalization of visual stimuli during eye movements) as determined in psychophysical studies in humans.

Tactile flow explains haptic counterparts of common visual illusions

Interaction with the external world requires the ability to perceive dynamic changes in complex sensorial input and react promptly. Here, we show that perception of dynamic stimuli in the visual and tactile sensory modalities share fundamental psychophysical aspects that can be explained by similar computational models. In vision, optic flow provides information on relative motion between the individual and the content of percept. For instance, radial patterns of optic flow are used to estimate time before contact with an approaching object [J.J. Gibson, What gives rise to the perception of motion? Psychol. Rev. 75 (1968) 335–346]. Similarly, in the tactile modality, radial patterns of stimuli provide information on softness of probed objects [A. Bicchi, D. De Rossi, E.P. Scilingo, The role of the contact area spread rate (CASR) in haptic discrimination of softness, IEEE Trans. Rob. Autom. 16 (2000) 496–504]. Optic flow is also invoked to explain several visual illusions, including the well-known “barber-pole” effect [N. Fisher, J.M. Zanker, The directional tuning of the barber-pole illusion, Perception 30 (2001) 1321–1336]. Here, we introduce a computational model of tactile flow, which is intimately related to existing models for the visual counterpart. The model accounts for psychophysical aspects of dynamic tactile perception and predicts illusory phenomena in the tactile domain, analogous to the barber-pole effect. When subjects touched translating pads with differently oriented gratings, they perceived a direction of motion that was significantly biased towards the orientation of the gratings. Therefore, these findings indicate that visual and tactile flow share similarities at the psychophysical and computational level and may be intended for similar perceptive goals. Results of this analysis have impact on the engineering of better haptic and multimodal interfaces for human–computer interaction.

The Effect of Visual Experience on the Development of Functional Architecture in hMT+

We investigated whether the visual hMT+ cortex plays a role in supramodal representation of sensory flow, not mediated by visual mental imagery. We used functional magnetic resonance imaging to measure neural activity in sighted and congenitally blind individuals during passive perception of optic and tactile flows. Visual motion-responsive cortex, including hMT+, was identified in the lateral occipital and inferior temporal cortices of the sighted subjects by response to optic flow. Tactile flow perception in sighted subjects activated the more anterior part of these cortical regions but deactivated the more posterior part. By contrast, perception of tactile flow in blind subjects activated the full extent, including the more posterior part. These results demonstrate that activation of hMT+ and surrounding cortex by tactile flow is not mediated by visual mental imagery and that the functional organization of hMT+ can develop to subserve tactile flow perception in the absence of any visual experience. Moreover, visual experience leads to a segregation of the motion-responsive occipitotemporal cortex into an anterior subregion involved in the representation of both optic and tactile flows and a posterior subregion that processes optic flow only.

Advances in Tracking the Features of a Geostatistical Field for Automatic Guidance

This paper is concerned with recognition and subsequent tracking of hardness patterns in coal deposita for the purpose of automatic guidance of a mining machine. A framework for the formal presentation of the problem and its sOlution is developed. An enhanced algorithm for determining tactile flow using linear correlation is presented. It has been shown that inclusion of only a single rule knowledge-base could improve the performance of the motion detection significantly. A trajectory learning scheme is suggested for acquiring the coal-seam undulation profile that is gradually matured as the mining progresses