Haptic Modality Research Articles

Telehaptic interfaces for interpersonal communication within a music ensemble

Visual communication is an important aspect of music performance, for example, to pick up temporal cues and find the right entries. Visual cues can also be instrumental to negotiate the solo order in improvized music or enable social exchange, for example, by signaling someone that her solo was well received. The problem with visual communication is that one has to catch someone else’s attention, and visual cues outside someone’s visual field cannot be detected, even more so if the addressee is busy reading a music score or closing his eyes in a Free Music session. Acoustic communication does not encounter these challenges, but of course someone does not want to disturb the music with other acoustic signals. The haptic modality has the advantage that it does not necessarily interfere with the acoustic signal and does not require attention. However, it allows interpersonal communication if both parties are within close proximity. Using telematic interfaces solves the problem of proximity by allowing participants to communicate over any physical distance. In the project presented here, haptic interfaces were explored in connection with an intelligent music system, CAIRA, to examine both the effect of human/machine and inter-human communication. [Work supported by the National Science Foundation, No. 1002851.]

Designing interaction with consumer products in a multisensory virtual reality environment

In the domain of industrial product development, products have been traditionally communicated to final users by means of technical drawings, sketches and physical prototypes. Recently, companies have tended to develop digital versions of products, which can be used with the purpose of communicating a new product to users, and also with the aim of allowing users to evaluate the product and its variants before its physical construction. Virtual prototyping is a relatively recent practice used in various industrial domains, which aims at anticipating a product that does not exist in reality yet. This practice can be used for evaluating the aesthetic quality of a product, its functional features and also its ergonomic and usability aspects. Current virtual reality technology well supports the implementation of effective virtual prototypes. In fact, one can touch, move, manipulate, and operate the virtual prototypes of products, such as household appliances, electronic devices, etc., with a good degree of realism. In order to do this, the interaction with virtual prototypes is multimodal and multisensory, and is often based on the combination of visual, haptic and auditory modalities. This paper shows how virtual prototypes can be used instead of physical artefacts or mock-ups for the communication of a new product or of variants of an existing product, and for the preliminary evaluation of its usage. The effectiveness of this practice is proved by tests performed by users.

Visual and Haptic Representations of Material Properties

Research on material perception has received an increasing amount of attention recently. Clearly, both the visual and the haptic sense play important roles in the perception of materials, yet it is still unclear how both senses compare in material perception tasks. Here, we set out to investigate the degree of correspondence between the visual and the haptic representations of different materials. We asked participants to both categorize and rate 84 different materials for several material properties. In the haptic case, participants were blindfolded and asked to assess the materials based on haptic exploration. In the visual condition, participants assessed the stimuli based on their visual impressions only. While categorization performance was less consistent in the haptic condition than in the visual one, ratings correlated highly between the visual and the haptic modality. PCA revealed that all material samples were similarly organized within the perceptual space in both modalities. Moreover, in both senses the first two principal components were dominated by hardness and roughness. These are two material features that are fundamental for the haptic sense. We conclude that although the haptic sense seems to be crucial for material perception, the information it can gather alone might not be quite fine-grained and rich enough for perfect material recognition.

The effect of vertical and horizontal symmetry on memory for tactile patterns in late blind individuals

Visual stimuli that exhibit vertical symmetry are easier to remember than stimuli symmetric along other axes, an advantage that extends to the haptic modality as well. Critically, the vertical symmetry memory advantage has not been found in early blind individuals, despite their overall superior memory, as compared with sighted individuals, and the presence of an overall advantage for identifying symmetric over asymmetric patterns. The absence of the vertical axis memory advantage in the early blind may depend on their total lack of visual experience or on the effect of prolonged visual deprivation. To disentangle this issue, in this study, we measured the ability of late blind individuals to remember tactile spatial patterns that were either vertically or horizontally symmetric or asymmetric. Late blind participants showed better memory performance for symmetric patterns. An additional advantage for the vertical axis of symmetry over the horizontal one was reported, but only for patterns presented in the frontal plane. In the horizontal plane, no difference was observed between vertical and horizontal symmetric patterns, due to the latter being recalled particularly well. These results are discussed in terms of the influence of the spatial reference frame adopted during exploration. Overall, our data suggest that prior visual experience is sufficient to drive the vertical symmetry memory advantage, at least when an external reference frame based on geocentric cues (i.e., gravity) is adopted.

Augmented visual, auditory, haptic, and multimodal feedback in motor learning: A review

It is generally accepted that augmented feedback, provided by a human expert or a technical display, effectively enhances motor learning. However, discussion of the way to most effectively provide augmented feedback has been controversial. Related studies have focused primarily on simple or artificial tasks enhanced by visual feedback. Recently, technical advances have made it possible also to investigate more complex, realistic motor tasks and to implement not only visual, but also auditory, haptic, or multimodal augmented feedback. The aim of this review is to address the potential of augmented unimodal and multimodal feedback in the framework of motor learning theories. The review addresses the reasons for the different impacts of feedback strategies within or between the visual, auditory, and haptic modalities and the challenges that need to be overcome to provide appropriate feedback in these modalities, either in isolation or in combination. Accordingly, the design criteria for successful visual, auditory, haptic, and multimodal feedback are elaborated.

Transfer of object category knowledge across visual and haptic modalities: Experimental and computational studies

We study people’s abilities to transfer object category knowledge across visual and haptic domains. If a person learns to categorize objects based on inputs from one sensory modality, can the person categorize these same objects when the objects are perceived through another modality? Can the person categorize novel objects from the same categories when these objects are, again, perceived through another modality? Our work makes three contributions. First, by fabricating Fribbles (3-D, multi-part objects with a categorical structure), we developed visual-haptic stimuli that are highly complex and realistic, and thus more ecologically valid than objects that are typically used in haptic or visual-haptic experiments. Based on these stimuli, we developed the See and Grasp data set, a data set containing both visual and haptic features of the Fribbles, and are making this data set freely available on the world wide web. Second, complementary to previous research such as studies asking if people transfer knowledge of object identity across visual and haptic domains, we conducted an experiment evaluating whether people transfer object category knowledge across these domains. Our data clearly indicate that we do. Third, we developed a computational model that learns multisensory representations of prototypical 3-D shape. Similar to previous work, the model uses shape primitives to represent parts, and spatial relations among primitives to represent multi-part objects. However, it is distinct in its use of a Bayesian inference algorithm allowing it to acquire multisensory representations, and sensory-specific forward models allowing it to predict visual or haptic features from multisensory representations. The model provides an excellent qualitative account of our experimental data, thereby illustrating the potential importance of multisensory representations and sensory-specific forward models to multisensory perception.

The perception and representation of orientations: A study in the haptic modality

This research examines the haptic perception of orientations in the frontal plane in order to identify the nature of their representation. Blindfolded participants inserted the tip of the index finger into a thimble mounted on the extremity of a haptic interface and manually explored the orientation of a “virtual rod”. After a short delay, participants had to reproduce the scanned orientation with the same hand without the guidance of the virtual rod. The analysis of the systematic errors showed that the recalled orientations were markedly biased toward the nearest diagonal in each quadrant with the exception of the orientations nearest to the vertical, which were biased toward the vertical. The variable error was greater for the oblique orientations than for the horizontal or vertical orientation. These results are interpreted with the Category-Adjustment model, which posits that orientations are categorically represented. We show that it is necessary to assume the existence of vertical and horizontal categories in addition to the previously postulated oblique categories to predict the error patterns observed in the present and former studies. The similarity of the error patterns in the visual and haptic modalities suggests that a common mechanism is at play in perceiving and reproducing orientations in both sensory modalities.

Need for touch and information processing strategies: An empirical examination

ABSTRACTDrawing from literatures on haptic modality and information processing, direct measures of information processing strategy are developed and applied to investigate whether touch search is a form of analytical processing. Applying two distinct approaches in the two studies, evidence of the processing style of those high in need for touch, compared with those low in need for touch, is gathered and tested in an online context where touch is not available. Cumulatively, the findings indicate that high‐need‐for‐touch consumers follow an analytical, feature‐by‐feature processing strategy, whereas those low in need for touch rely more on a relational processing strategy. Theoretical and managerial implications are provided. Copyright © 2012 John Wiley & Sons, Ltd.

The haptic recognition of geometrical shapes in congenitally blind and blindfolded adolescents: is there a haptic prototype effect?

BackgroundIt has been shown that visual geometrical shape categories (rectangle and triangle) are graded structures organized around a prototype as demonstrated by perception and production tasks in adults as well as in children. The visual prototypical shapes are better recognized than other exemplars of the categories. Their existence could emerge from early exposure to these prototypical shapes that are present in our visual environment. The present study examined the role of visual experience in the existence of prototypical shapes by comparing the haptic recognition of geometrical shapes in congenitally blind and blindfolded adolescents.Methodology/Principal FindingsTo determine whether the existence of a prototype effect (higher recognition of prototypical shapes than non prototypical shapes) depended on visual experience, congenitally blind and blindfolded sighted adolescents were asked to recognize in the haptic modality three categories of correct shapes (square, rectangle, triangle) varying in orientation (prototypical/canonical orientation vs. non prototypical/canonical orientation rotated by 45°) among a set of other shapes. A haptic prototype effect was found in the blindfolded sighted whereas no difference between prototypical and non prototypical correct shapes was observed in the congenitally blind. A control experiment using a similar visual recognition task confirmed the existence of a visual prototype effect in a group of sighted adolescents.Conclusion/SignificanceThese findings show that the prototype effect is not intrinsic to the haptic modality but depends on visual experience. This suggests that the occurrence of visual and haptic prototypical shapes in the recognition of geometrical shape seems to depend on visual exposure to these prototypical shapes existing in our environment.

Exploratory procedures of tactile images in visually impaired and blindfolded sighted children: How they relate to their consequent performance in drawing

The aim of the present study was to compare the types of exploratory procedures employed by children when exploring bidimensional tactile patterns and correlate the use of these procedures with the children's shape drawing performance. 18 early blind children, 20 children with low vision and 24 age-matched blindfolded sighted children aged approximately 7 or 11 years were included in the study. The children with a visual handicap outperformed the sighted children in terms of haptic exploration and did not produce less recognizable drawings than their sighted counterparts. Close relationships were identified between the types of exploratory procedures employed by the children and their subsequent drawing performance, regardless of visual status. This close link between action and perception in the haptic modality indicates the importance of training blind children in exploratory procedures at an early age.

The influence of different sensory cues as selection feedback and co-location in presence and task performance

For some applications based on virtual reality technology, presence and task performance are important factors to validate the experience. Different approaches have been adopted to analyse the extent to which certain aspects of a computer-generated environment may enhance these factors, but mainly in 2D graphical user interfaces. This study explores the influence of different sensory modalities on performance and the sense of presence experienced within a 3D environment. In particular, we have evaluated visual, auditory and active haptic feedback for indicating selection of virtual objects. The effect of spatial alignment between proprioceptive and visual workspaces (co-location) has also been analysed. An experiment has been made to evaluate the influence of these factors in a controlled 3D environment based on a virtual version of the Simon game. The main conclusions obtained indicate that co-location must be considered in order to determine the sensory needs during interaction within a virtual environment. This study also provides further evidence that the haptic sensory modality influences presence to a higher extent, and that auditory cues can reduce selection times. Conclusions obtained provide initial guidelines that will help designers to set out better selection techniques for more complex environments, such as training simulators based on VR technology, by highlighting different optimal configurations of sensory feedback.

Haptic Communications

Audiovisual communications is at the core of multimedia systems that allow users to interact across distances. It is common understanding that both audio and video are required for high-quality interaction. While audiovisual information provides a user with a satisfactory impression of being present in a remote environment, physical interaction and manipulation is not supported. True immersion into a distant environment and efficient distributed collaboration require the ability to physically interact with remote objects and to literally get in touch with other people. Touching and manipulating objects remotely becomes possible if we augment traditional audiovisual communications by the haptic modality. Haptic communications is a relatively young field of research that has the potential to substantially improve human-human and human-machine interaction. In this paper, we discuss the state-of-the-art in haptic communications both from psychophysical and technical points of view. From a human perception point of view, we mainly focus on the multimodal integration of video and haptics and the improved performance that can be achieved when combining them. We also discuss how the human adapts to discrepancies and synchronization errors between different modalities, a research area which is typically referred to as perceptual learning. From a technical perspective, we address perceptual coding of haptic information and the transmission of haptic data streams over resource-constrained and potentially lossy networks in the presence of unpredictable and time-varying communication delays. In this context, we also discuss the need for objective quality metrics for haptic communication. Throughout the paper, we stress the fact that haptic communications is not meant as a replacement of traditional audiovisual communications but rather as an additional dimension for telepresence that will allow us to advance in our quest for truly immersive communication.

Vibrotactile music systems for co-located and telematic performance

Building on previous findings, the project reported here expands on the idea of how the modality of touch contributes to the sensory perception of sound in the presence of vibro-tactile events. The SenseAble1.0, a vibrotactile interface, was developed to transform the auditory parameter space into one that is adequate for haptic perception by the means of eight individually controllable actuators and machine learning algorithms. Due to the process of sensory substitution, people with hearing impairments can use the vibrotactile display to “listen” to music and perceive exterior acoustic stimuli through the sense of touch. A psychophysical pilot study investigated (i) frequency discrimination and pitch direction as well as (ii) interval size recognition of the haptic modality, and (iii) adaptation effects of the somatosensory system. Live performances using the SenseAble as a vibrotactile music system illustrate to what extent musical communication between ensemble members and a hearing-impaired musician i...

Haptic cues as a utility to perceive and recognise geometry

Research has been conducted on how to aid blind peoples' perceptions and cognition of scientific data and, specifically, on how to strengthen their background in mathematics as a means of accomplishing this goal. In search of alternate modes to vision, researchers and practitioners have studied the opportunities of haptics alone and in combination with other modes, such as audio. What is already known, and has motivated research in this area, is that touch and vision might form a common brain representation that is shared between the visual and haptic modalities and through haptics learning is active rather than passive. In spite of extensive research on haptics in the areas of psychology and neuropsychology, recent advances and rare experiences in using haptic technology have not caused a transfer from basic knowledge in the area of haptics to learning applications and practical guidelines on how to develop such applications. Thus motivated, this study investigates different haptic effects, such as free space, magnetic effects and the bounded box when blind people are given the task of recognising and manipulating classes of 3D objects with which they have varying familiarity. In parallel, this study investigates the applicability of Sjostrom's guidelines on haptic applications development and uses his problem classification to capture knowledge from the experiments. The results of this study show that users can easily recognise and manipulate familiar objects, albeit with some assistance. There is an indication that users completed tasks faster and needed less assistance with magnetic effects. However, they were not as satisfied with this mode. While the results of this study show that haptics have the potential to allow students to conceptualise 3D objects, much more work is needed to exploit this technology to the fullest. Objects with higher complexity are difficult for students, and, in their opinion, the virtual objects (as presented) leave much room for improvement. Sjostrom's error taxonomy proved useful, and four of five sub-guidelines tested were confirmed to be useful in this study.

Virtual Rehabilitation Environment Using Principles of Intrinsic Motivation and Game Design

This paper presents a novel multimodal virtual rehabilitation environment. Its design and implementation are based on principles related to intrinsic motivation and game design. The system consists of visual, acoustic, and haptic modalities. Elements contributing to intrinsic motivation are carefully joined in the three modalities to increase patients' motivation during the long process of rehabilitation. The message in a bottle (MIB) virtual scenario is designed to allow interplay between motor and cognitive challenges in the exercising patient. The user first needs to perform a motor action to receive a cognitive challenge that is finally solved by a second motor action. Visual feedback provides the most relevant information related to the task. Acoustic feedback consists of environmental sounds, music, and spoken instructions or encouraging statements for the patient. The haptic modality generates tactile information related to the environment and provides various modes of assistance for the patient's arm movements. The MIB scenario was evaluated with 16 stroke patients, who rated it positively using the Intrinsic Motivation Inventory questionnaire. Additionally, the MIB scenario seems to elicit higher motivation than a simpler pick-and-place training task.

Effect of kinesthetic and tactile haptic feedback on the quality of experience of edutainment applications

Haptic technologies and applications have received enormous attention in the last decade. The incorporation of haptic modality into multimedia applications adds excitement and enjoyment to an application. It also adds a more natural feel to multimedia applications, that otherwise would be limited to vision and audition, by engaging as well the user's sense of touch, giving a more intrinsic feel essential for ambient intelligent applications. However, the improvement of an application's Quality of Experience (QoE) by the addition of haptic feedback is still not completely understood. The research presented in this paper focuses on the effect of haptic feedback and what it potentially adds to the experience of the user as opposed to the traditional visual and auditory feedback. In essence, it investigates certain issues regarding stylus-based haptic education applications and haptic-enhanced entertainment videos. To this end, we used two haptic applications: the haptic handwriting learning tool to experiment with force feedback haptic interaction and the tactile YouTube application for tactile haptic feedback. In both applications, our analysis shows that the addition of haptic feedback will increase the QoE in the absence of fatigue or discomfort for this category of applications. This implies that the incorporation of haptic modality (both force feedback as well as tactile feedback) has positively contributed to the overall QoE for the users.

Visual Influence on Haptic Torque Perception

The brain receives input from multiple sensory modalities simultaneously, yet we experience the outside world as a single integrated percept. This integration process must overcome instances where perceptual information conflicts across sensory modalities. Under such conflicts, the relative weighting of information from each modality typically depends on the given task. For conflicts between visual and haptic modalities, visual information has been shown to influence haptic judgments of object identity, spatial features (e.g., location, size), texture, and heaviness. Here we test a novel instance of haptic-visual conflict in the perception of torque. We asked participants to hold a left-right unbalanced object while viewing a potentially left-right mirror-reversed image of the object. Despite the intuition that the more proximal haptic information should dominate the perception of torque, we find that visual information exerts substantial influences on torque perception even when participants know that visual information is unreliable.

Categorizing natural objects: a comparison of the visual and the haptic modalities

Although the hands are the most important tool for humans to manipulate objects, only little is known about haptic processing of natural objects. Here, we selected a unique set of natural objects, namely seashells, which vary along a variety of object features, while others are shared across all stimuli. To correctly interact with objects, they have to be identified or categorized. For both processes, measuring similarities between objects is crucial. Our goal is to better understand the haptic similarity percept by comparing it to the visual similarity percept. First, direct similarity measures were analyzed using multidimensional scaling techniques to visualize the perceptual spaces of both modalities. We find that the visual and the haptic modality form almost identical perceptual spaces. Next, we performed three different categorization tasks. All tasks exhibit a highly accurate processing of complex shapes of the haptic modality. Moreover, we find that objects grouped into the same category form regions within the perceptual space. Hence, in both modalities, perceived similarity constitutes the basis for categorizing objects. Moreover, both modalities focus on shape to form categories. Taken together, our results lead to the assumption that the same cognitive processes link haptic and visual similarity perception and the resulting categorization behavior.

Eco-Driving Performance Assessment With in-Car Visual and Haptic Feedback Assistance

In this experiment, 28 participants completed an urban driving task in a highly immersive driving simulator at Renault’s Technical Centre for Simulation. This simulator provides a 150 deg field of view in a fully instrumented cockpit. Two different eco-driving assistance devices were added: a visual display on the midconsole and a force feedback system on the gas pedal, in order to apply an additionnal reaction torque on drivers’ foot. The feedback information was computed by comparing the car’s instant acceleration with an optimal acceleration level based on a proprietary consumption model of a Renault diesel engine. This experiment has three main goals: I. Assess the contribution of verbal instructions to eco-driving performance; II. Quantify the additional contribution generated by two eco-driving assistance systems (visual and haptic); III. Measure drivers’ acceptance of haptic eco-driving assistance system. Basic eco-driving instructions, such as changing gears under 2000 Rpm, yield significant decrease of polluting emissions. Assisting drivers with visual, haptic, or visual-haptic on-board devices, in addition to low engine speed verbal instructions, lead to supplementary significant savings of polluting emissions. There is no significant difference between assistance feedback type; suggesting that the haptic feedback provides the same ecoperformance as visual feedback. In particular, subjects show good adaptation to the haptic feedback pedal at first utilization of the system. They apparently relied more on haptic modality to achieve the eco-driving task, when they used both visual and haptic assistance.

Similarity and categorization: From vision to touch

Even though human perceptual development relies on combining multiple modalities, most categorization studies so far have focused on the visual modality. To better understand the mechanisms underlying multisensory categorization, we analyzed visual and haptic perceptual spaces and compared them with human categorization behavior. As stimuli we used a three-dimensional object space of complex, parametrically-defined objects. First, we gathered similarity ratings for all objects and analyzed the perceptual spaces of both modalities using multidimensional scaling analysis. Next, we performed three different categorization tasks which are representative of every-day learning scenarios: in a fully unconstrained task, objects were freely categorized, in a semi-constrained task, exactly three groups had to be created, whereas in a constrained task, participants received three prototype objects and had to assign all other objects accordingly. We found that the haptic modality was on par with the visual modality both in recovering the topology of the physical space and in solving the categorization tasks. We also found that within-category similarity was consistently higher than across-category similarity for all categorization tasks and thus show how perceptual spaces based on similarity can explain visual and haptic object categorization. Our results suggest that both modalities employ similar processes in forming categories of complex objects.