Vision-based Force Estimation Research Articles

Vision-Based Force Estimation for Minimally Invasive Telesurgery Through Contact Detection and Local Stiffness Models

Vision-Based Force Estimation for Minimally Invasive Telesurgery Through Contact Detection and Local Stiffness Models

Vision-based estimation of manipulation forces by deep learning of laparoscopic surgical images obtained in a porcine excised kidney experiment

In robot-assisted surgery, in which haptics should be absent, surgeons experience haptics-like sensations as “pseudo-haptic feedback”. As surgeons who routinely perform robot-assisted laparoscopic surgery, we wondered if we could make these “pseudo-haptics” explicit to surgeons. Therefore, we created a simulation model that estimates manipulation forces using only visual images in surgery. This study aimed to achieve vision-based estimations of the magnitude of forces during forceps manipulation of organs. We also attempted to detect over-force, exceeding the threshold of safe manipulation. We created a sensor forceps that can detect precise pressure at the tips with three vectors. Using an endoscopic system that is used in actual surgery, images of the manipulation of excised pig kidneys were recorded with synchronized force data. A force estimation model was then created using deep learning. Effective detection of over-force was achieved if the region of the visual images was restricted by the region of interest around the tips of the forceps. In this paper, we emphasize the importance of limiting the region of interest in vision-based force estimation tasks.

Force estimation from 4D OCT data in a human tumor xenograft mouse model

Abstract Minimally invasive robotic surgery offer benefits such as reduced physical trauma, faster recovery and lesser pain for the patient. For these procedures, visual and haptic feedback to the surgeon is crucial when operating surgical tools without line-of-sight with a robot. External force sensors are biased by friction at the tool shaft and thereby cannot estimate forces between tool tip and tissue. As an alternative, vision-based force estimation was proposed. Here, interaction forces are directly learned from deformation observed by an external imaging system. Recently, an approach based on optical coherence tomography and deep learning has shown promising results. However, most experiments are performed on ex-vivo tissue. In this work, we demonstrate that models trained on dead tissue do not perform well in in vivo data. We performed multiple experiments on a human tumor xenograft mouse model, both on in vivo, perfused tissue and dead tissue. We compared two deep learning models in different training scenarios. Training on perfused, in vivo data improved model performance by 24% for in vivo force estimation.

Generalization of spatio-temporal deep learning for vision-based force estimation

Abstract Robot-assisted minimally-invasive surgery is increasingly used in clinical practice. Force feedback offers potential to develop haptic feedback for surgery systems. Forces can be estimated in a vision-based way by capturing deformation observed in 2D-image sequences with deep learning models. Variations in tissue appearance and mechanical properties likely influence force estimation methods’ generalization. In this work, we study the generalization capabilities of different spatial and spatio-temporal deep learning methods across different tissue samples. We acquire several data-sets using a clinical laparoscope and use both purely spatial and also spatio-temporal deep learning models. The results of this work show that generalization across different tissues is challenging. Nevertheless, we demonstrate that using spatio-temporal data instead of individual frames is valuable for force estimation. In particular, processing spatial and temporal data separately by a combination of a ResNet and GRU architecture shows promising results with a mean absolute error of 15.450 compared to 19.744 mN of a purely spatial CNN.

Better grip force control by attending to the controlled object: Evidence for direct force estimation from visual motion

Estimating forces acting between our hand and objects is essential for dexterous motor control. An earlier study suggested that vision contributes to the estimation by demonstrating changes in grip force pattern caused by delayed visual feedback. However, two possible vision-based force estimation processes, one based on hand position and another based on object motion, were both able to explain the effect. Here, to test each process, we examined how visual feedback of hand and object each contribute to grip force control during moving an object (mass) connected to the grip by a damped-spring. Although force applied to the hand could be estimated from its displacement, we did not find any improvements by the hand feedback. In contrast, we found that visual feedback of object motion significantly improved the synchrony between grip and load forces. Furthermore, when both feedback sources were provided, the improvement was observed only when participants were instructed to direct their attention to the object. Our results suggest that visual feedback of object motion contributes to estimation of dynamic forces involved in our actions by means of inverse dynamics computation, i.e., the estimation of force from motion, and that visual attention directed towards the object facilitates this effect.

Sensory Substitution for Force Feedback Recovery

Robotic-assisted surgeries are commonly used today as a more efficient alternative to traditional surgical options. Both surgeons and patients benefit from those systems, as they offer many advantages, including less trauma and blood loss, fewer complications, and better ergonomics. However, a remaining limitation of currently available surgical systems is the lack of force feedback due to the teleoperation setting, which prevents direct interaction with the patient. Once the force information is obtained by either a sensing device or indirectly through vision-based force estimation, a concern arises on how to transmit this information to the surgeon. An attractive alternative is sensory substitution, which allows transcoding information from one sensory modality to present it in a different sensory modality. In the current work, we used visual feedback to convey interaction forces to the surgeon. Our overarching goal was to address the following question: How should interaction forces be displayed to support efficient comprehension by the surgeon without interfering with the surgeon’s perception and workflow during surgery? Until now, the use the visual modality for force feedback has not been carefully evaluated. For this reason, we conducted an experimental study with two aims: (1) to demonstrate the potential benefits of using this modality and (2) to understand the surgeons’ perceptual preferences. The results derived from our study of 28 surgeons revealed a strong positive acceptance of the users (96%) using this modality. Moreover, we found that for surgeons to easily interpret the information, their mental model must be considered, meaning that the design of the visualizations should fit the perceptual and cognitive abilities of the end user. To our knowledge, this is the first time that these principles have been analyzed for exploring sensory substitution in medical robotics. Finally, we provide user-centered recommendations for the design of visual displays for robotic surgical systems.

Vision-based sensing of forces in elastic objects

A minimally intrusive, vision-based, computational force sensor for elastically deformable objects is proposed in this paper. Estimating forces from the visually measured displacements is straightforward in the case of the linear problem of small displacements, but not in the case of the large displacements where geometric non-linearities must be taken into account. From the images of the object taken before and after the deformation, we compute the deformation gradients and logarithmic strains. Using the stress–strain relationships for the material, we compute the Cauchy’s stresses and from this we estimate the locations and magnitudes of the external forces that caused the deformation. A sensitivity analysis is performed to examine the effect of small deviations in the experimentally captured displacements on the estimated external forces. This analysis showed that the small-strain case is more sensitive and prone to numerical errors than the large-strain case. Additionally, a related method that is indirect and iterative is also presented in which we assume that we know the locations of the external forces. Numerical and experimental studies are presented for both micro- and macro-scale objects. The main conclusion of this work is that the vision-based force estimation is viable if the displacements of the deforming object can be captured accurately.