Robotic Eye Research Articles

Review on challenges for robotic eye surgery; surgical systems, technologies, cost-effectiveness, and controllers.

In recent decades, a number of surgical systems have been developed and are applied for a growing variety of surgeries. This review will consider the significant challenges of robotic surgery for the eye. These challenges take into account the different eye diseases, available technologies, and costs in different surgical systems for the eye. The conditions of a suitable controller will be discussed with consideration of relevant control engineering concepts. Comparison is made between the different characteristics of surgical robots for the eye. In this review, some comparisons will be made in eye surgical robots, control algorithms, sensors in surgical robots, communication protocols, and actuators.

Simultaneous Online Registration-Independent Stiffness Identification and Tip Localization of Surgical Instruments in Robot-assisted Eye Surgery.

Notable challenges during retinal surgery lend themselves to robotic assistance which has proven beneficial in providing a safe steady-hand manipulation. Efficient assistance from the robots heavily relies on accurate sensing of surgery states (e.g. instrument tip localization and tool-to-tissue interaction forces). Many of the existing tool tip localization methods require preoperative frame registrations or instrument calibrations. In this study using an iterative approach and by combining vision and force-based methods, we develop calibration- and registration-independent (RI) algorithms to provide online estimates of instrument stiffness (least squares and adaptive). The estimations are then combined with a state-space model based on the forward kinematics (FWK) of the Steady-Hand Eye Robot (SHER) and Fiber Brag Grating (FBG) sensor measurements. This is accomplished using a Kalman Filtering (KF) approach to improve the deflected instrument tip position estimations during robot-assisted eye surgery. The conducted experiments demonstrate that when the online RI stiffness estimations are used, the instrument tip localization results surpass those obtained from pre-operative offline calibrations for stiffness.

Robotics and cybersurgery in ophthalmology: a current perspective.

Ophthalmology is one of the most enriched fields, allowing the domain of artificial intelligence to be part of its point of interest in scientific research. The requirement of specialized microscopes and visualization systems presents a challenge to adapting robotics in ocular surgery. Cyber-surgery has been used in other surgical specialties aided by Da Vinci robotic system. This study focuses on the current perspective of using robotics and cyber-surgery in ophthalmology and highlights factors limiting their progression. A review of literature was performed with the aid of Google Scholar, Pubmed, CINAHL, MEDLINE (N.H.S. Evidence), Cochrane, AMed, EMBASE, PsychINFO, SCOPUS, and Web of Science. Keywords: Cybersurgery, Telesurgery, ophthalmology robotics, Da Vinci robotic system, artificial intelligence in ophthalmology, training on robotic surgery, ethics of the use of robots in medicine, legal aspects, and economics of cybersurgery and robotics. 150 abstracts were reviewed for inclusion, and 68 articles focusing on ophthalmology were included for full-text review. Da Vinci Surgical System has been used to perform a pterygium repair in humans and was successful in ex vivo corneal, strabismus, amniotic membrane, and cataract surgery. Gamma Knife enabled effective treatment of uveal melanoma. Robotics used in ophthalmology were: Da Vinci Surgical System, Intraocular Robotic Interventional Surgical System (IRISS), Johns Hopkins Steady-Hand Eye Robot and smart instruments, and Preceyes' B.V. Cybersurgery is an alternative to overcome distance and the shortage of surgeons. However, cost, availability, legislation, and ethics are factors limiting the progression of these fields. Robotic and cybersurgery in ophthalmology are still in their niche. Cost-effective studies are needed to overcome the delay. Technologies, such as 5G and Tactile Internet, are required to help reduce resource scheduling problems in cybersurgery. In addition, prototype development and the integration of artificial intelligence applications could further enhance the safety and precision of ocular surgery.

Modified Hand–Eye Calibration Using Dual Quaternions

This paper presents a modified model for hand–eye calibration based on dual quaternion algebra. By using dual quaternions to represent the rotations and translations of a rigid body simultaneously in the task space, the formulation is elegant for the analysis of the hand–eye equation. The hand–eye transformation derived in this study is represented in a compact manner, which uses a combination of the dual part and the real part of the dual quaternion. Although the hand–eye equation can be solved by using six elements of a dual quaternion without using its scalar parts, the scaler numbers in both the real and dual parts of a dual quaternion contain part of the pose information. The originality is based on the derivation of the construct of the identification algorithm of external parameters of the camera by using all eight elements of a dual quaternion. Then, the data transformation between the cameras of the dual-arm hand–eye robot system is presented. The corresponding results demonstrate that the proposed hand–eye calibration algorithm can process measurement data with noise and can also improve the identification accuracy to verify its efficiency.

Effects of predictive robot eyes on attentional processes in HRI

Effects of predictive robot eyes on attentional processes in HRI

Artificial intelligence in healthcare: Public perception of robotic surgery

Abstract Breakthrough after breakthrough, artificial intelligence has proven itself to be of support to the healthcare system. Public and private hospitals are improving their services through the adoption of different technologies, such as wearables, 3D printing, virtual reality or surgical robots, among others. This paper investigates the public opinion on one of the healthcare areas affected by the evolution of artificial intelligence, the robotic eye surgery (RES). The assessment is done from the safety, technology and pricing perspectives, considering the demographic distribution of the sample. The results reveal that RES is considered a safer alternative to traditional procedures performed by surgeons alone. The technology’s influence over the medical field is seen as a significant contributor to public trust in RES. The price evolution for such procedures in the following years is seen with slight optimism. The demographic variables used in the paper have a weak or no relationship with public trust in RES, the opinion on its price evolution or the view over technology’s impact on the medical field. Understanding the present view over robotic surgery is valuable for developing support measures, awareness campaigns and adjusting the healthcare providers’ decisions.

Design of an optical sensor system for robots inspired by the human eye

The development of autonomous motion capability by robotic systems, particularly in dynamic environments, is strongly related to the sensor systems installed in the robot. An important feature of these sensors should be their ability to detect the world in the same way as humans do. This paper focuses on the design of a prototype of a robotic eye, or optical sensor, which has functional characteristics similar to those of the human eye and complies with two Degrees of Freedom (DOF) in its movement. In addition to this, the entire design process of the parts is documented with computer-aided design (CAD) support, the manipulation software, and the programming structure used to control it. The performance tests performed and the analysis of the results obtained demonstrate the total fulfillment of the proposed objectives, which were even surpassed.

Anthropomorphic Robotic Eyes: Structural Design and Non-Verbal Communication Effectiveness.

This paper shows the structure of a mechanical system with 9 DOFs for driving robot eyes, as well as the system’s ability to produce facial expressions. It consists of three subsystems which enable the motion of the eyeballs, eyelids, and eyebrows independently to the rest of the face. Due to its structure, the mechanical system of the eyeballs is able to reproduce all of the motions human eyes are capable of, which is an important condition for the realization of binocular function of the artificial robot eyes, as well as stereovision. From a kinematic standpoint, the mechanical systems of the eyeballs, eyelids, and eyebrows are highly capable of generating the movements of the human eye. The structure of a control system is proposed with the goal of realizing the desired motion of the output links of the mechanical systems. The success of the mechanical system is also rated on how well it enables the robot to generate non-verbal emotional content, which is why an experiment was conducted. Due to this, the face of the human-like robot MARKO was used, covered with a face mask to aid in focusing the participants on the eye region. The participants evaluated the efficiency of the robot’s non-verbal communication, with certain emotions achieving a high rate of recognition.

Delta Robot Kinematic Calibration for Precise Robot-Assisted Retinal Surgery.

High precision is required for ophthalmic robotic systems. This paper presents the kinematic calibration for the delta robot which is part of the next generation of Steady-Hand Eye Robot (SHER). A linear error model is derived based on geometric error parameters. Two experiments with different ranges of workspace are conducted with laser sensors measuring displacement. The error parameters are identified and applied in the kinematics to compensate for modeling error. To achieve better accuracy, Bernstein polynomials are adopted to fit the error residuals after compensation. After the kinematic calibration process, the error residuals of the delta robot are reduced to satisfy the clinical requirements.

Humans Can't Resist Robot Eyes - Reflexive Cueing With Pseudo-Social Stimuli.

Joint attention is a key mechanism for humans to coordinate their social behavior. Whether and how this mechanism can benefit the interaction with pseudo-social partners such as robots is not well understood. To investigate the potential use of robot eyes as pseudo-social cues that ease attentional shifts we conducted an online study using a modified spatial cueing paradigm. The cue was either a non-social (arrow), a pseudo-social (two versions of an abstract robot eye), or a social stimulus (photographed human eyes) that was presented either paired (e.g. two eyes) or single (e.g. one eye). The latter was varied to separate two assumed triggers of joint attention: the social nature of the stimulus, and the additional spatial information that is conveyed only by paired stimuli. Results support the assumption that pseudo-social stimuli, in our case abstract robot eyes, have the potential to facilitate human-robot interaction as they trigger reflexive cueing. To our surprise, actual social cues did not evoke reflexive shifts in attention. We suspect that the robot eyes elicited the desired effects because they were human-like enough while at the same time being much easier to perceive than human eyes, due to a design with strong contrasts and clean lines. Moreover, results indicate that for reflexive cueing it does not seem to make a difference if the stimulus is presented single or paired. This might be a first indicator that joint attention depends rather on the stimulus' social nature or familiarity than its spatial expressiveness. Overall, the study suggests that using paired abstract robot eyes might be a good design practice for fostering a positive perception of a robot and to facilitate joint attention as a precursor for coordinated behavior.

Human Preferences for Robot Eye Gaze in Human-to-Robot Handovers.

This paper investigates human’s preferences for a robot’s eye gaze behavior during human-to-robot handovers. We studied gaze patterns for all three phases of the handover process: reach, transfer, and retreat, as opposed to previous work which only focused on the reaching phase. Additionally, we investigated whether the object’s size or fragility or the human’s posture affect the human’s preferences for the robot gaze. A public data-set of human-human handovers was analyzed to obtain the most frequent gaze behaviors that human receivers perform. These were then used to program the robot’s receiver gaze behaviors. In two sets of user studies (video and in-person), a collaborative robot exhibited these gaze behaviors while receiving an object from a human. In the video studies, 72 participants watched and compared videos of handovers between a human actor and a robot demonstrating each of the three gaze behaviors. In the in-person studies, a different set of 72 participants physically performed object handovers with the robot and evaluated their perception of the handovers for the robot’s different gaze behaviors. Results showed that, for both observers and participants in a handover, when the robot exhibited Face-Hand-Face gaze (gazing at the giver’s face and then at the giver’s hand during the reach phase and back at the giver’s face during the retreat phase), participants considered the handover to be more likable, anthropomorphic, and communicative of timing (p < 0.0001). However, we did not find evidence of any effect of the object’s size or fragility or the giver’s posture on the gaze preference.

Underwater Fish Detection and Counting Using Mask Regional Convolutional Neural Network

Fish production has become a roadblock to the development of fish farming, and one of the issues encountered throughout the hatching process is the counting procedure. Previous research has mainly depended on the use of non-machine learning-based and machine learning-based counting methods and so was unable to provide precise results. In this work, we used a robotic eye camera to capture shrimp photos on a shrimp farm to train the model. The image data were classified into three categories based on the density of shrimps: low density, medium density, and high density. We used the parameter calibration strategy to discover the appropriate parameters and provided an improved Mask Regional Convolutional Neural Network (Mask R-CNN) model. As a result, the enhanced Mask R-CNN model can reach an accuracy rate of up to 97.48%.

Insights on embodiment induced by visuo-tactile stimulation during robotic telepresence

Using a simple neuroscience-inspired procedure to beam human subjects into robots, we previously demonstrated by visuo-motor manipulations that embodiment into a robot can enhance the acceptability and closeness felt towards the robot. In that study, the feelings of likeability and closeness toward the robot were significantly related to the sense of agency, independently of the sensations of enfacement and location. Here, using the same paradigm we investigated the effect of a purely sensory manipulation on the sense of robotic embodiment associated to social cognition. Wearing a head-mounted display, participants saw the visual scene captured from the robot eyes. By positioning a mirror in front of the robot, subjects saw themselves as a robot. Tactile stimulation was provided by stroking synchronously or not with a paintbrush the same location of the subject and robot faces. In contrast to the previous motor induction of embodiment which particularly affected agency, tactile induction yields more generalized effects on the perception of ownership, location and agency. Interestingly, the links between positive social feelings towards the robot and the strength of the embodiment sensations were not observed. We conclude that the embodiment into a robot is not sufficient in itself to induce changes in social cognition.

Towards a Clinically Optimized Tilt Mechanism for Bilateral Micromanipulation with Steady-Hand Eye Robot.

Cooperative robotic systems for vitreoretinal surgery can enable novel surgical approaches by allowing the surgeon to perform procedures with enhanced stabilization and high accuracy tool movements. This paper presents the optimization and design of a four-bar linkage type tilt mechanism for a novel Steady-Hand Eye Robot (SHER) which can be used equivalently on both, the left and right patient side, during a bilateral approach with two robots. In this optimization, it is desirable to limit the workspace needed for compensation motions that ensure a virtual remote center of motion (V-RCM). The safety space around the patient, the space for the surgeon's hand and maintaining positional accuracy are also included in the optimization. The applicability of the resulting optimized mechanism was confirmed with a design prototype in a representative mock-up of the surgical setting allowing multiple directions of robot approach towards a medical phantom.

Spotlight-based 3D Instrument Guidance for Autonomous Task in Robot-assisted Retinal Surgery.

Retinal surgery is known to be a complicated and challenging task for an ophthalmologist even for retina specialists. Image guided robot-assisted intervention is among the novel and promising solutions that may enhance human capabilities during microsurgery. In this paper, a novel method is proposed for 3D navigation of a microsurgical instrument based on the projection of a spotlight during robot-assisted retinal surgery. To test the feasibility and effectiveness of the proposed method, a vessel tracking task in a phantom with a Remote Center of Motion (RCM) constraint is performed by the Steady-Hand Eye Robot (SHER). The results are compared to manual tracking, cooperative control tracking with the SHER and spotlight-based automatic tracking with SHER. The reported results are that the spotlight-based automatic tracking with SHER can reach an average tracking error of 0.013 mm and keeping distance error of 0.1 mm from the desired range demonstrating a significant improvement compared to manual or cooperative control methods alone.

Artificial Eyes with Emotion and Light Responsive Pupils for Realistic Humanoid Robots

This study employs a novel 3D engineered robotic eye system with dielectric elastomer actuator (DEA) pupils and a 3D sculpted and colourised gelatin iris membrane to replicate the appearance and materiality of the human eye. A camera system for facial expression analysis (FEA) was installed in the left eye, and a photo-resistor for measuring light frequencies in the right. Unlike previous prototypes, this configuration permits the robotic eyes to respond to both light and emotion proximal to a human eye. A series of experiments were undertaken using a pupil tracking headset to monitor test subjects when observing positive and negative video stimuli. A second test measured pupil dilation ranges to high and low light frequencies using a high-powered artificial light. This data was converted into a series of algorithms for servomotor triangulation to control the photosensitive and emotive pupil dilation sequences. The robotic eyes were evaluated against the pupillometric data and video feeds of the human eyes to determine operational accuracy. Finally, the dilating robotic eye system was installed in a realistic humanoid robot (RHR) and comparatively evaluated in a human-robot interaction (HRI) experiment. The results of this study show that the robotic eyes can emulate the average pupil reflex of the human eye under typical light conditions and to positive and negative emotive stimuli. However, the results of the HRI experiment indicate that replicating natural eye contact behaviour was more significant than emulating pupil dilation.

Visual Saliency Estimation through Manifold Learning

Saliency detection has been a desirable way for robotic vision to find the most noticeable objects in a scene. In this paper, a robust manifold-based saliency estimation method has been developed to help capture the most salient objects in front of robotic eyes, namely cameras. In the proposed approach, an image is considered as a manifold of visual signals (stimuli) spreading over a connected grid, and local visual stimuli are compared against the global image variation to model the visual saliency. With this model, manifold learning is then applied to minimize the local variation while keeping the global contrast, and turns the RGB image into a multi-channel image. After the projection through manifold learning, histogram-based contrast is then computed for saliency modeling of all channels of the projected images, and mutual information is introduced to evaluate each single-channel saliency map against prior knowledge to provide cues for the fusion of multiple channels. In the last step, the fusion procedure combines all single-channel saliency maps according to their mutual information score, and generates the final saliency map. In our experiment, the proposed method is evaluated using one of the largest publicly available image datasets. The experimental results demonstrate that our algorithm consistently outperforms the state-of-the-art unsupervised saliency detection methods, yielding higher precision and better recall rates. Furthermore, the proposed method is tested on a video-type test dataset where a moving camera is trying to catch up with the walking person---a salient object in the video sequence. Our experimental results show that the proposed approach can successful accomplish this task, revealing its potential use for similar robotic applications.

Two degree-of-freedom robotic eye: design, modeling, and learning-based control in foveation and smooth pursuit

With increasing ocular motility disorders affecting human eye movement, the need to understand the biomechanics of the human eye rises constantly. A robotic eye system that physically mimics the human eye can serve as a useful tool for biomedical researchers to obtain an intuitive understanding of the functions and defects of the extraocular muscles and the eye. This paper presents the design, modeling, and control of a two degree-of-freedom (2-DOF) robotic eye, driven by artificial muscles, in particular, made of super-coiled polymers (SCPs). Considering the highly nonlinear dynamics of the robotic eye system, this paper applies deep deterministic policy gradient (DDPG), a machine learning algorithm to solve the control design problem in foveation and smooth pursuit of the robotic eye. To the best of our knowledge, this paper presents the first modeling effort to establish the dynamics of a robotic eye driven by SCP actuators, as well as the first control design effort for robotic eyes using a DDPG-based control strategy. A linear quadratic regulator-type reward function is proposed to achieve a balance between system performances (convergence speed and tracking accuracy) and control efforts. Simulation results are presented to demonstrate the effectiveness of the proposed control strategy for the 2-DOF robotic eye.

A review of robotic surgical training: establishing a curriculum and credentialing process in ophthalmology.

Ophthalmic surgery requires a highly dexterous and precise surgical approach to work within the small confines of the eye, and the use of robotics offers numerous potential advantages to current surgical techniques. However, there is a lag in the development of a comprehensive training and credentialing system for robotic eye surgery, and certification of robotic skills proficiency relies heavily on industry leadership. We conducted a literature review on the curricular elements of established robotics training programs as well as privileging guidelines from various institutions to outline key components in training and credentialing robotic surgeons for ophthalmic surgeries. Based on our literature review and informal discussions between the authors and other robotic ophthalmic experts, we recommend that the overall training framework for robotic ophthalmic trainees proceeds in a stepwise, competency-based manner from didactic learning, to simulation exercises, to finally operative experiences. Nontechnical skills such as device troubleshooting and interprofessional teamwork should also be formally taught and evaluated. In addition, we have developed an assessment tool based on validated global rating scales for surgical skills that may be used to monitor the progress of trainees. Finally, we propose a graduating model for granting privileges to robotic surgeons. Further work will need to be undertaken to assess the feasibility, efficacy and integrity of the training curriculum and credentialing practices for robotic ophthalmic surgery.

Eye Machines: Robot Eye, Vision and Gaze

Eye Machines: Robot Eye, Vision and Gaze