Haptic Input Device Research Articles

Impact Localization for Haptic Input Devices Using Hybrid Laminates with Sensoric Function

The required energy savings can be achieved in all automotive domains through weight savings and the merging of manufacturing processes in production. This fact is taken into account through functional integration in lightweight materials and manufacturing in a process close to large-scale production. In previous work, separate steps of a process chain for manufacturing a center console cover utilizing a sensoric hybrid laminate have been developed and evaluated. This includes the process steps of joining, forming and inline polarization as well as connecting to an embedded system. This work continues the research process by evaluating impact localization methods to use the center console as a haptic input device. For this purpose, different deep learning methods are derived from the state of the art and analyzed for their applicability in two consecutive studies. The results show that MLPs, LSTMs, GRUs and CNNs are suitable to localize impacts on the novel laminate with high localization rates of up to 99%, and thus the usability of the developed laminate as a haptic input device has been proven.

Development and evaluation of a simulator for endoscopic sinus surgery

Endoscopic surgical procedures have been established as gold standard in sinus surgery. Challenges for surgical training have been addressed by the use of virtual reality (VR) simulators. To date, a number of simulators have been developed. However, previous studies regarding their training effects investigated only medically pretrained subjects or the time course of training outcomes has not been reported. A computer tomography (CT) dataset was segmented manually. A three-dimensional polygonal surface model was generated and textured using original photographic material. Interaction with the virtual environment was performed using a haptic input device. For the investigation of training outcomes with the simulator, the parameters duration and the number of errors were recorded. Ten subjects completed a training consisting of five runs on ten consecutive days. Within the whole exercise period, four subjects reduced the duration of intervention by more than 60%. Four subjects reduced the number of errors by more than 60%. Eight out of 10 subjects showed an improvement with respect to both parameters. On median, the duration of the procedure was reduced by 46 seconds and the number of errors by 191. The statistical analysis between the two parameters showed a positive correlation. Our data suggests that training on the FESS-simulator considerably improves the performance even in inexperienced subjects, both in terms of duration and accuracy of the procedure.

Exodex Adam-A Reconfigurable Dexterous Haptic User Interface for the Whole Hand.

Applications for dexterous robot teleoperation and immersive virtual reality are growing. Haptic user input devices need to allow the user to intuitively command and seamlessly “feel” the environment they work in, whether virtual or a remote site through an avatar. We introduce the DLR Exodex Adam, a reconfigurable, dexterous, whole-hand haptic input device. The device comprises multiple modular, three degrees of freedom (3-DOF) robotic fingers, whose placement on the device can be adjusted to optimize manipulability for different user hand sizes. Additionally, the device is mounted on a 7-DOF robot arm to increase the user’s workspace. Exodex Adam uses a front-facing interface, with robotic fingers coupled to two of the user’s fingertips, the thumb, and two points on the palm. Including the palm, as opposed to only the fingertips as is common in existing devices, enables accurate tracking of the whole hand without additional sensors such as a data glove or motion capture. By providing “whole-hand” interaction with omnidirectional force-feedback at the attachment points, we enable the user to experience the environment with the complete hand instead of only the fingertips, thus realizing deeper immersion. Interaction using Exodex Adam can range from palpation of objects and surfaces to manipulation using both power and precision grasps, all while receiving haptic feedback. This article details the concept and design of the Exodex Adam, as well as use cases where it is deployed with different command modalities. These include mixed-media interaction in a virtual environment, gesture-based telemanipulation, and robotic hand–arm teleoperation using adaptive model-mediated teleoperation. Finally, we share the insights gained during our development process and use case deployments.

A Digital Twin Approach for Contextual Assistance for Surgeons During Surgical Robotics Training.

Minimally invasive robotic surgery copes with some disadvantages for the surgeon of minimally invasive surgery while preserving the advantages for the patient. Most commercially available robotic systems are telemanipulated with haptic input devices. The exploitation of the haptics channel, e.g., by means of Virtual Fixtures, would allow for an individualized enhancement of surgical performance with contextual assistance. However, it remains an open field of research as it is non-trivial to estimate the task context itself during a surgery. In contrast, surgical training allows to abstract away from a real operation and thus makes it possible to model the task accurately. The presented approach exploits this fact to parameterize Virtual Fixtures during surgical training, proposing a Shared Control Parametrization Engine that retrieves procedural context information from a Digital Twin. This approach accelerates a proficient use of the robotic system for novice surgeons by augmenting the surgeon’s performance through haptic assistance. With this our aim is to reduce the required skill level and cognitive load of a surgeon performing minimally invasive robotic surgery. A pilot study is performed on the DLR MiroSurge system to evaluate the presented approach. The participants are tasked with two benchmark scenarios of surgical training. The execution of the benchmark scenarios requires basic skills as pick, place and path following. The evaluation of the pilot study shows the promising trend that novel users profit from the haptic augmentation during training of certain tasks.

Cyber-physical-system for representing a robot end effector

Programming by Demonstration (PbD) is a method to program robots through the performance of a task by humans. Most implementations are online methods that use visual or force feedback of the demonstrator. However, we developed an offline programming approach for PbD with a special input device within an Augmented Reality Environment. Therefore, this paper aims to answer how the characteristics and functionality of the end effector of a jointed-arm robot can be represented by a haptic input device in order to perform PbD. The PbD process is first carried out on a digital twin of the robot, visualized to the user in real physical space by means of augmented reality technology. The programming of the digital twin can later be transferred to the real robot. The haptic input device in this context is the main part of the Cyber-Physical-System (CPS), which enables the user to interact with the virtual robot. Therefore, the specification of the mechanical and software components of the CPS is of main importance. Within this paper, strategies for the implementation of shape and function abstraction, as well as for ensuring communication, have been worked out. The physical shape of the CPS is kept generic and is only subject to ergonomic restrictions. However, Augmented Reality overlays the physical shape with an exact digital image of the end effector used later in the process. Nevertheless, the physical characteristics of the real robot should be represented as real as possible by the CPS. Therefore, the CPS is equipped with various sensors and actuators. With the CPS it is possible to determine contact forces and to manipulate objects to a certain extent in order to teach gripping strategies to the digital twin. An operating system was developed for communication and control of the electronic components. For the validation of the functionality of the CPS an exemplary PbD process was developed, the results were analyzed and evaluated.

Hybrid Laminate for Haptic Input Device with Integrated Signal Processing

Achieving lightweight construction through only material substitution does not realize the full potential of producing a lightweight material, hence, it is no longer sufficient. Weight-saving goals are best achieved through additional function integration. In order to implement this premise for mass production, a manufacturing process for joining and forming hybrid laminates using a new tool concept is presented. All materials used are widely producible and processable. The manufactured cover of an automotive center console serves to demonstrate a human interface device with impact detection and action execution. This is only possible through a machine learning system, which is implemented on a small—and thus space- and energy-saving—embedded system. The measurement results confirm the objective and show that localization was sufficiently accurate.

Haptic and visual augmented reality interface for programming welding robots

It is a challenging task for operators to program a remote robot for welding manipulation depending only on the visual information from the remote site. This paper proposes an intuitive user interface for programming welding robots remotely using augmented reality (AR) with haptic feedback. The proposed system uses a depth camera to reconstruct the surfaces of workpieces. A haptic input device is used to allow users to define welding paths along these surfaces. An AR user interface is developed to allow users to visualize and adjust the orientation of the welding torch. Compared with the traditional robotic welding path programming methods which rely on prior CAD models or contact between the robot end-effector and the workpiece, this proposed approach allows for fast and intuitive remote robotic welding path programming without prior knowledge of CAD models of the workpieces. The experimental results show that the proposed approach is a user-friendly interface and can assist users in obtaining an accurate welding path.

Direct manipulation is better than passive viewing for learning anatomy in a three-dimensional virtual reality environment

With the advancement of virtual reality (VR) technologies, medical students may now study complex anatomical structures in three-dimensional (3-D) virtual environments, without relying solely upon high cost, unsustainable cadavers or animal models. When coupled with a haptic input device, these systems support direct manipulation and exploration of the anatomical structures. Yet, prior studies provide inconclusive support for direct manipulation beyond passive viewing in virtual environments. In some cases, exposure to an “optimal view” appears to be the main source of learning gains, regardless of participants’ control of the system. In other cases, direct manipulation provides benefits beyond passive viewing. To address this issue, we compared medical students who either directly manipulated a virtual anatomical structure (inner ear) or passively viewed an interaction in a stereoscopic, 3-D environment. To ensure equal exposure to optimal views we utilized a yoked-pair design, such that for each participant who manipulated the structure a single matched participant viewed a recording of this interaction. Results indicate that participants in the manipulation group were more likely to successful generate (i.e., draw) the observed structures at posttest than the viewing group. Moreover, manipulation benefited students with low spatial ability more than students with high spatial ability. These results suggest that direct manipulation of the virtual environment facilitated embodiment of the anatomical structure and helped participants maintain a clear frame of reference while interacting, which particularly supported participants with low spatial ability.

Development of a compact continuum tubular robotic system for nasopharyngeal biopsy

Traditional posterior nasopharyngeal biopsy using a flexible nasal endoscope has the risks of abrasion and injury to the nasal mucosa and thus causing trauma to the patient. Recently, a new class of robots known as continuum tubular robots (CTRs) provide a novel solution to the challenge with miniaturized size, curvilinear maneuverability, and capability of avoiding collision within the nasal environment. This paper presents a compact CTR which is 35 cm in total length, 10 cm in diameter, 2.15 kg in weight, and easy to be integrated with a robotic arm to perform more complicated operations. Structural design, end-effector design, and workspace analysis are described in detail. In addition, teleoperation of the CTR using a haptic input device is developed for position control in 3D space. Moreover, by integrating the robot with three electromagnetic tracking sensors, a navigation system together with a shape reconstruction algorithm is developed. Comprehensive experiments are conducted to test the functionality of the proposed prototype; experiment results show that under teleoperation, the system has an accuracy of 2.20 mm in following a linear path, an accuracy of 2.01 mm in following a circular path, and a latency time of 0.1 s. It is also found that the proposed shape reconstruction algorithm has a mean error of around 1 mm along the length of the tubes. Besides, the feasibility and effectiveness of the proposed robotic system being applied to posterior nasopharyngeal biopsy are demonstrated by a cadaver experiment. The proposed robotic system holds promise to enhance clinical operation in transnasal procedures.

Haptic User Interface for a Telerobotic System – Design based on a Network Model

Loss of haptic feedback is one major drawback of telerobotic surgery systems. For this reason tasks of passive input devices can be taken over by haptic user interfaces. The main objective of this paper is the development of a haptic user interface with a kinematic structure matching the slave robot's kinematic structure. In addition, we focus on the detailed design and characterization of a haptic input device concerning the grasping degree of freedom. Requirements of the user interface are derived by analyzing the medical application and the structure of the slave robot. Since the user interface and the input device are supposed to provide realistic haptic feedback, static and dynamic behavior of the device is of interest. Electromechanical analogies help to design the dynamic behavior of the haptic input device. A characterization of static and dynamic behavior validates our design process. The implementation of the input device in the overall user interface is shown.The designed user interface, allows controlling the slave robot in three dimensions, rotation along the axis of the main kinematic chain and grasping. The user interfaces kinematic structure is taking the kinematic constrains of the slave kinematic into account. The input device can be handled similar as a forceps with three point precision grip, similar to a common surgical procedure without robotic assistance. The user interface provides haptic feedback in all directions of the kinematic structure.

The removable prosthetic restorations utilizing CAD/CAM system

Recently, the digital solution of fabricating removable prosthesis by applying haptic input device, electronic surveying, and rapid prototyping was introduced. This review article covers the concept of electronic surveying, computer-aided denture framework designing procedure, discussions after several digital denture cases, directions of future development, such as digital tooth arrangement and RP flasking.

Universal life: the use of virtual worlds among people with disabilities

With their emphasis on 3D graphics and complex interface controls, it would appear that virtual worlds have little to offer to people with disabilities. On the contrary, multi-user virtual environments, such as Linden Lab's Second Life platform, serve as a form of augmented reality where users transcend physiological or cognitive challenges to great social and therapeutic benefit. A number of intriguing developments exist within the accessibility sector, making barrier-free access an important aspect of the interaction experience. Examples include haptic input devices for the blind, virtual regions developed according to Universal Design principles, communities dedicated to people with cognitive disorders, the use of the avatar as counselor, and customizable personae that either transcend or represent a disabled person's self-identity. This paper investigates research methods and case studies affiliated with virtual environments, as well as the ways inclusive design removes barriers to access for users with disabilities.

A haptic-based part decomposition method for multi-material product design

Multi-material products are getting increasingly popular in recent years. Yet no systems have been developed to support the design of such products. In this article, a haptic-based method for designing multi-material products is proposed. The proposed method consists of three main parts: haptic painting/marking, boundary smoothing/fitting, and volume decomposition. A prototype system based on the three parts has been implemented using a haptic input device. The haptic device provides an intuitive user interface for quick volume mark-up in a multi-material product by direct mesh painting. Each painted region represents a material volume whose boundary can be automatically traced. The boundary points are then used as control points for Catmull-Rom spline fitting. To constrain a spline to the original mesh surface, the spline is projected onto the mesh. Now, using the boundary splines, volume decomposition can be done automatically when needed. Each decomposed volume can be assigned a different material or colour. The numerous iterations of volume mark-up and decomposition in the early stage of multi-material product design can now be made easy and effective using the proposed method.

Dual-task performance in multimodal human-computer interaction: a psychophysiological perspective

This paper examines the psychophysiological effects of mental workload in single-task and dual-task human-computer interaction. A mental arithmetic task and a manual error correction task were performed both separately and concurrently on a computer using verbal and haptic input devices. Heart rate, skin conductance, respiration and peripheral skin temperature were recorded in addition to objective performance measures and self-report questionnaires. Analysis of psychophysiological responses found significant changes from baseline for both single-task and dual-task conditions. There were also significant psychophysiological differences between the mental arithmetic task and the manual error correction task, but no differences in questionnaire results. Additionally, there was no significant psychophysiological difference between performing only the mental arithmetic task and performing both tasks at once. These findings suggest that psychophysiological measures respond differently to different types of tasks and that they do not always agree with performance or with participants' subjective feelings.

MiroSurge—Advanced User Interaction Modalities in Minimally Invasive Robotic Surgery

This paper presents MiroSurge, a telepresence system for minimally invasive surgery developed at the German Aerospace Center (DLR), and introduces MiroSurge's new user interaction modalities: (1) haptic feedback with software-based preservation of the fulcrum point, (2) an ultrasound-based approach to the quasi-tactile detection of pulsating vessels, and (3) a contact-free interface between surgeon and telesurgery system, where stereo vision is augmented with force vectors at the tool tip. All interaction modalities aim to increase the user's perception beyond stereo imaging by either augmenting the images or by using haptic interfaces. MiroSurge currently provides surgeons with two different interfaces. The first option, bimanual haptic interaction with force and partial tactile feedback, allows for direct perception of the remote environment. Alternatively, users can choose to control the surgical instruments by optically tracked forceps held in their hands. Force feedback is then provided in augmented stereo images by constantly updated force vectors displayed at the centers of the teleoperated instruments, regardless of the instruments' position within the video image. To determine the centerpoints of the instruments, artificial markers are attached and optically tracked. A new approach to detecting pulsating vessels beneath covering tissue with an omnidirectional ultrasound Doppler sensor is presented. The measurement results are computed and can be provided acoustically (by displaying the typical Doppler sound), optically (by augmenting the endoscopic video stream), or kinesthetically (by a gentle twitching of the haptic input devices). The control structure preserves the fulcrum point in minimally invasive surgery and user commands are followed by the surgical instrument. Haptic feedback allows the user to distinguish between interaction with soft and hard environments. The paper includes technical evaluations of the features presented, as well as an overview of the system integration of MiroSurge.

Haptic Feedback in the Instructional Environment and its Relationship to Visual Attention and Learning

This study, involving a computer-based training application coupled with a haptic input device, offered insight into how haptic feedback influences visual attention in learning applications. Data was collected from 33 students (ages 11–14) who completed a learning module on lever principles. This data consists of paper-based pre- and post-test scores, embedded module test scores and eye movement data. Embedded test scores were significantly higher in the non-haptic condition. Eye movement data revealed differential influences of haptic feedback. A path analysis of gaze movements indicated that haptic feedback was associated with increased frequencies of movements between zones of interest surrounding lever components but there were no significant increases in the frequencies of eye movements between the levers and the numeric force readout. Implications of these findings are discussed.

The Effects of Simulated Inertia and Force Prediction on Delayed Telepresence

This paper presents two methods aimed at alleviating the negative effects of network delays on teleoperation. The problem of telepresence across delayed networks is well known. A delay in feedback information such as visual and haptic data can make the task at hand very unintuitive and difficult for the operator. The first presented method investigates the hypothesis that simulated inertia in the haptic input device can be a supporting factor during teleoperation across delayed networks. An experiment involving 36 human subjects was carried out under varying network and inertia conditions. Psychophysical experiments were conducted to determine suitable values of inertia. However, simulated inertia was found to be neither a supporting factor nor a detrimental factor to operator performance and immersion in the presence of both delayed and non-delayed networks. The second presented method is a force prediction approach, which extends the teleoperation system with a local force model. This is a learned force model situated locally at the operator-side. Instead of relying on the delayed force signals from the teleoperator-side, haptic information can be extracted from this local force model. An experiment has been created to demonstrate the benefits of this approach in compensating for the instabilities due to time delay.

A novel haptic platform for real time bilateral biomanipulation with a MEMS sensor for triaxial force feedback

A novel triaxial force sensing device developed by the authors with a MEMS sensor as core component is mounted on a subnanometric resolution nanomanipulator having three degrees of freedom (DOF). This sensorized device allows measuring forces on the nanomanipulator tip in the range of 0–3 N for normal and ±50 mN for tangential forces with a resolution of 11 bits. Together with a haptic input device, a setup was created allowing palpation and force feeling. The mathematical model used to drive the master haptic interface force feedback capabilities is based on online force and stiffness measurement. The performance of the novel setup is demonstrated with a cell palpation experiment.

Development of a high-performance haptic telemanipulation system with dissimilar kinematics

This work addresses selected practical issues regarding the development of a telerobotic system for 6-d.o.f. tasks. The system consists of a hyper-redundant 10-d.o.f. haptic input device ViSHaRD10, a redundant 7-d.o.f. manipulator and a stereo vision system. The redundancy of the haptic input device is exploited to assure large convex workspaces and singularity-free operation. The anthropomorphic construction of the telemanipulator enables intuitive manipulation and increases the user-friendliness of the overall system. As a practical benchmark, an assembly experiment in 6 d.o.f. for the case of a negligible time delay was performed. Issues regarding inverse kinematics, spatial interaction control, transparency and intuitiveness of teleoperation are discussed.

HAPTIC DEVICES AS NEW MECHATRONIC COMPONENTS FOR INTERACTIVE GRID APPLICATIONS

New mechanical haptic input devices for surgical training and a modular electrical control interface have been developed. The interface has a modular design and is accessed via USB or internet, respectively. Special virtual reality simulation models were designed for the evaluation of the new hardware. Clinical evaluation studies have been carried out. We present a new approach towards interactive applications for the grid network.