Robotic Eye Research Articles

Mentalistic attention orienting triggered by android eyes

The eyes play a special role in human communications. Previous psychological studies have reported reflexive attention orienting in response to another individual’s eyes during live interactions. Although robots are expected to collaborate with humans in various social situations, it remains unclear whether robot eyes have the potential to trigger attention orienting similarly to human eyes, specifically based on mental attribution. We investigated this issue in a series of experiments using a live gaze-cueing paradigm with an android. In Experiment 1, the non-predictive cue was the eyes and head of an android placed in front of human participants. Light-emitting diodes in the periphery served as target signals. The reaction times (RTs) required to localize the valid cued targets were faster than those for invalid cued targets for both types of cues. In Experiment 2, the gaze direction of the android eyes changed before the peripheral target lights appeared with or without barriers that made the targets non-visible, such that the android did not attend to them. The RTs were faster for validly cued targets only when there were no barriers. In Experiment 3, the targets were changed from lights to sounds, which the android could attend to even in the presence of barriers. The RTs to the target sounds were faster with valid cues, irrespective of the presence of barriers. These results suggest that android eyes may automatically induce attention orienting in humans based on mental state attribution.

Enhancing social robot's direct gaze expression through vestibulo-ocular movements

Non-verbal communication, especially eye contact, plays a crucial role in human interaction. Integrating eye contact capabilities into robotic behavior presents challenges that require comprehensive interdisciplinary collaboration from the fields of vision science and robotics. This study introduces an innovative methodology to improve the user's perception of the tabletop social robot Haru's direct gaze by explaining how eye and head movements can be managed through simulating visual perception processing. By considering the mechanisms of eye and head movements during visual perception processing, the robot's eye and head gaze behaviors are intended to replicate human gaze movements with Vestibulo-Ocular Movement (VOM) during face-to-face interaction, ultimately enhancing the user's perception of the robot's direct gaze.

Releaf: An Efficient Method for Real-Time Occlusion Handling by Game Theory.

Receiving uninterrupted videos from a scene with multiple cameras is a challenging task. One of the issues that significantly affects this task is called occlusion. In this paper, we propose an algorithm for occlusion handling in multi-camera systems. The proposed algorithm, which is called Real-time leader finder (Releaf), leverages mechanism design to assign leader and follower roles to each of the cameras in a multi-camera setup. We assign leader and follower roles to the cameras and lead the motion by the camera with the least occluded view using the Stackelberg equilibrium. The proposed approach is evaluated on our previously open-sourced tendon-driven 3D-printed robotic eye that tracks the face of a human subject. Experimental results demonstrate the superiority of the proposed algorithm over the Q-leaning and Deep Q Networks (DQN) baselines, achieving an improvement of 20% and 18% for horizontal errors and an enhancement of 81% for vertical errors, as measured by the root mean squared error metric. Furthermore, Releaf has the superiority of real-time performance, which removes the need for training and makes it a promising approach for occlusion handling in multi-camera systems.

Investigation of Joint Action in Go/No-Go Tasks: Development of a Human-Like Eye Robot and Verification of Action Space

AbstractHuman–robot collaboration (HRC) is a natural progression of technological development and can improve job performance, address labor shortages, and reduce labor costs. However, it is still uncertain whether joint action, similar to that occurring between humans, can be replicated between humans and robots. Many robotic researchers have focused on joint action, and it has been demonstrated that gaze cueing plays a significant role in this context. Currently, previous studies on joint action use humanoids; however, robots utilized in the research on human-robot collaboration lack human-like eyes needed for verification. Therefore, this study focuses on the development of an eye robot with gaze-cueing behaviors that can be easily integrated into existing robotic systems. As another theme of this study, we proposed the use of fixation duration as a new metric, which is distinct from the commonly used response time, for the quantitative evaluation of joint action research. These are verified through a Go/No-go task under six conditions—three behavioral (i.e., joint action, joint attention-only, and alone), each with two partner conditions (robot or human partner). While developing a human-like eye robot, this study demonstrates the potential of a robot to be a better joint action partner than an uncertain human, with participants exhibiting the best reaction times when partnered with a robot. The shared action space of the participants was investigated, where a transference of the action space indicates the expression of joint action. The fixation duration indicates that the proposed robot cause participants to move their action space to include that of the robot. These results suggest that the proposed collaborative robot can initiate a joint action between a robot and a human, and can perform as a more effective partner in joint actions compared to an unfamiliar human. This study showcased the capacity of fixation duration as a quantitative assessment metric for joint action.

Humanoid robot eyeballs driven by bubble artificial muscles

Currently, service robots play an increasingly significant role in human-computer interaction, and have their intelligent, humanized, emotional, and expressive capabilities have constantly improved. Eye movement is essential for humanoid service robot facial expression, which may significantly improve human-computer connection. This project aims to create bionic robot eyes powered by novel bubble artificial muscles (BAMs), resulting in low-cost and straightforward robot eyes with horizontal, vertical, and mixed ocular movement capabilities. By adjusting the air pressure input to the BAMs, an antagonistic mechanism composed of a BAM and a tension spring is meant to move an eyeball horizontally or vertically. The results of an experiment reveal that humanoid robot eyeballs can imitate human eye movement. This research will contribute to the development of a low-weight, low-cost humanoid service robot eyeball drive system that can realise the expression of robot facial expression by controlling the movement of robot eyeball, thereby enabling humanoid servers to effectively express their feelings and emotions in accordance with human needs and enhancing user interaction.

Emergence of Human Oculomotor Behavior in a Cable-Driven Biomimetic Robotic Eye Using Optimal Control

Emergence of Human Oculomotor Behavior in a Cable-Driven Biomimetic Robotic Eye Using Optimal Control

When Trustworthiness Meets Face: Facial Design for Social Robots.

As a technical application in artificial intelligence, a social robot is one of the branches of robotic studies that emphasizes socially communicating and interacting with human beings. Although both robot and behavior research have realized the significance of social robot design for its market success and related emotional benefit to users, the specific design of the eye and mouth shape of a social robot in eliciting trustworthiness has received only limited attention. In order to address this research gap, our study conducted a 2 (eye shape) × 3 (mouth shape) full factorial between-subject experiment. A total of 211 participants were recruited and randomly assigned to the six scenarios in the study. After exposure to the stimuli, perceived trustworthiness and robot attitude were measured accordingly. The results showed that round eyes (vs. narrow eyes) and an upturned-shape mouth or neutral mouth (vs. downturned-shape mouth) for social robots could significantly improve people's trustworthiness and attitude towards social robots. The effect of eye and mouth shape on robot attitude are all mediated by the perceived trustworthiness. Trustworthy human facial features could be applied to the robot's face, eliciting a similar trustworthiness perception and attitude. In addition to empirical contributions to HRI, this finding could shed light on the design practice for a trustworthy-looking social robot.

Realistic 3D human saccades generated by a 6-DOF biomimetic robotic eye under optimal control.

We recently developed a biomimetic robotic eye with six independent tendons, each controlled by their own rotatory motor, and with insertions on the eye ball that faithfully mimic the biomechanics of the human eye. We constructed an accurate physical computational model of this system, and learned to control its nonlinear dynamics by optimising a cost that penalised saccade inaccuracy, movement duration, and total energy expenditure of the motors. To speed up the calculations, the physical simulator was approximated by a recurrent neural network (NARX). We showed that the system can produce realistic eye movements that closely resemble human saccades in all directions: their nonlinear main-sequence dynamics (amplitude-peak eye velocity and duration relationships), cross-coupling of the horizontal and vertical movement components leading to approximately straight saccade trajectories, and the 3D kinematics that restrict 3D eye orientations to a plane (Listing's law). Interestingly, the control algorithm had organised the motors into appropriate agonist-antagonist muscle pairs, and the motor signals for the eye resembled the well-known pulse-step characteristics that have been reported for monkey motoneuronal activity. We here fully analyse the eye-movement properties produced by the computational model across the entire oculomotor range and the underlying control signals. We argue that our system may shed new light on the neural control signals and their couplings within the final neural pathways of the primate oculomotor system, and that an optimal control principle may account for a wide variety of oculomotor behaviours. The generated data are publicly available at https://data.ru.nl/collections/di/dcn/DSC_626870_0003_600.

Cooperative vs. Teleoperation Control of the Steady Hand Eye Robot with Adaptive Sclera Force Control: A Comparative Study.

A surgeon's physiological hand tremor can significantly impact the outcome of delicate and precise retinal surgery, such as retinal vein cannulation (RVC) and epiretinal membrane peeling. Robot-assisted eye surgery technology provides ophthalmologists with advanced capabilities such as hand tremor cancellation, hand motion scaling, and safety constraints that enable them to perform these otherwise challenging and high-risk surgeries with high precision and safety. Steady-Hand Eye Robot (SHER) with cooperative control mode can filter out surgeon's hand tremor, yet another important safety feature, that is, minimizing the contact force between the surgical instrument and sclera surface for avoiding tissue damage cannot be met in this control mode. Also, other capabilities, such as hand motion scaling and haptic feedback, require a teleoperation control framework. In this work, for the first time, we implemented a teleoperation control mode incorporated with an adaptive sclera force control algorithm using a PHANTOM Omni haptic device and a force-sensing surgical instrument equipped with Fiber Bragg Grating (FBG) sensors attached to the SHER 2.1 end-effector. This adaptive sclera force control algorithm allows the robot to dynamically minimize the tool-sclera contact force. Moreover, for the first time, we compared the performance of the proposed adaptive teleoperation mode with the cooperative mode by conducting a vessel-following experiment inside an eye phantom under a microscope.

Simultaneously Calibration of Multi Hand–Eye Robot System Based on Graph

Precise calibration is the basis for the vision-guided robot system to achieve high-precision operations. Systems with multi-eyes (cameras) and multi-hands (robots) are particularly sensitive to calibration errors. Most existing methods focus on the calibration of a single unit of the whole system, such as poses between hand and eye, or between two hands. These methods can be used to determine the pose between each unit, but the serialized incremental calibration strategy cannot avoid the error accumulation problem in a large-scale system. Instead of focusing on a single unit, this paper models the multi-eye and multi-hand system calibration problem as a graph and proposes a method based on the minimum spanning tree and graph optimization. This method can automatically plan the serialized optimal calibration strategy in accordance with the system settings to get coarse calibration results initially. Then, with these initial values, the closed-loop constraints are introduced to carry out global optimization with different sensor's accuracy considered. As a general calibration method, it can be applied to different multi-robot systems. Simulation experiments demonstrate the performance of the proposed algorithm under different noises and various hand-eye configurations. In addition, experiments on real robot systems are presented to further verify the proposed method.

Seeing through a robot's eyes: A cross-sectional exploratory study in developing a robotic screening technology for autism.

The present exploratory cross-sectional case-control study sought to develop a reliable and scalable screening tool for autism using a social robot. The robot HUMANE, installed with computer vision and linked with recognition technology, detected the direction of eye gaze of children. Children aged 3-8 (M = 5.52; N = 199) participated, 87 of whom had been confirmed with autism, 55 of whom were suspected to have autism, and 57 of whom were not considered to cause any concern for having autism. Before a session, a human experimenter instructed HUMANE to narrate a story to a child. HUMANE prompted the child to return his/her eye gaze to the robot if the child looked away, and praised the child when it re-established its eye gaze quickly after a prompt. The reliability of eye gaze detection was checked across all pairs of human raters and HUMANE and reached 0.90, indicating excellent interrater agreement. Using the pre-specified reference standard (Autism Spectrum Quotient), the sensitivity and specificity of the index tests (i.e., the number of robot prompts and duration of inattentiveness) reached 0.88 or above and the Diagnostic Odds Ratios were beyond 190. These results show that social robots may detect atypical eye patterns, suggesting a potential future for screening autism using social robots.

Intelligent Surveillance Using Robot Eye and Sensor System

Majority of the advancements, which have been made in the field of robotics are employed in household applications. Surveillance systems are used in guarding and security applications of household and workplaces. This paper presents an advanced surveillance system which integrates sensor system, robot technology and intelligent programming. Robotic surveillance system can be used to replace human guards in dangerous surveillance conditions. Several robots have been developed to assist in these operations. But the full scale employment of robots in crucial fields such as surveillance, security in defence industries and military bases is very limited. The system's overall structure is described, along with descriptions of each component. The robotic system consists of robot eyes, which are capable of sensing an object through IR sensor capability. This robotic system helps in the identification of human presence and alerting respective owners. These operations, performed by the robotic system are achieved using sensors, several number of servo motors, Arduino microcontroller. These components are placed in their respective locations, within a body which is fabricated using the 3D printing technology. The sensors in the robotic system capture the movements of objects. The robotic eyes are fitted with cameras which capture the movements. The robotic system is programmed in such a way that the robotic eyes, move in the direction of object which has been sensed by the sensors and focus on it. If there are multiple moving objects, then one robotic eye focuses on one object, while the other robotic eye focuses on another object. The captured surveillance footage can be viewed through a monitor connected to the robotic system.

Eye-mounting goggles to bridge the gap between benchtop experiments and in vivo robotic eye surgery

A variety of robot-assisted surgical systems have been proposed to improve the precision of eye surgery. Evaluation of these systems has typically relied on benchtop experiments with artificial or enucleated eyes. However, this does not properly account for the types of head motion that are common among patients undergoing eye surgery, which a clinical robotic system will encounter. In vivo experiments are clinically realistic, but they are risky and thus require the robotic system to be at a sufficiently mature state of development. In this paper, we describe a low-cost device that enables an artificial or enucleated eye to be mounted to standard swim goggles worn by a human volunteer to enable more realistic evaluation of eye-surgery robots after benchtop studies and prior to in vivo studies. The mounted eye can rotate about its center, with a rotational stiffness matching that of an anesthetized patient’s eye. We describe surgeon feedback and technical analyses to verify that various aspects of the design are sufficient for simulating a patient’s eye during surgery.

My Eyes Speak: Improving Perceived Sociability of Autonomous Vehicles in Shared Spaces Through Emotional Robotic Eyes

The ability of autonomous vehicles (AVs) to interact socially with pedestrians poses a significant impact on their integration with urban traffic. This is particularly important for vehicle-pedestrian shared spaces due to increased social requirements in comparison to vehicular roads. Current pedestrian experience in shared spaces suffers from negative attitudes towards AVs and the consequently low acceptability of AVs in these spaces. HRI work shows that the acceptability of robots in public spaces can be positively impacted by their perceived sociability (i.e., possessing social skills), which can be enhanced by their ability to express emotions. Inspired by this approach, we follow a systematic process to design emotional expressions for AVs using the headlight ("eye'') area and investigate their impact on perceived sociability of AVs in shared spaces, by conducting expert focus groups (N=12) and an online video-based user study (N=106). Our findings confirm that the perceived sociability of AVs can be enhanced by emotional expressions indicated through emotional eyes. We further discuss implications of our findings for improving pedestrian experience and attitude in shared spaces and highlight opportunities to use AVs' emotional expressions as a new external communication strategy for future research.

Design, Implementation, and Observer-based Output Control of a Super-coiled Polymer-Driven Two Degree-of-Freedom Robotic Eye.

The prevalence of ineffective corrective surgeries for ocular motor disorders calls for a robotic eye platform in aiding ophthalmologists to better understand the biomechanisms of human eye movement. This letter presents the first hardware design and implementation of a 2-DOF robotic eye driven by super-coiled polymer (SCP) artificial muscles. While our previous work designed and simulated a deep deterministic policy gradient (DDPG) learning-based controller that requires full-state feedback of the SCP-driven robotic eye, measuring the temperature states of the slender SCPs is generally impractical for the ubiquitously aimed robot. To address this predicament, this letter proposes a reduced-order state observer to estimate the temperature of SCPs given the kinematic measurements. Combining the designed observer and the learning-based controller, the closed-loop output feedback control is implemented on the robotic eye prototype to examine its performance on three classical types of eye movements: visual fixation, saccadic pursuit, and smooth pursuit. The experimental results are presented which successfully validate the observer-based output control of the SCP-driven robotic eye.

The potential of robot eyes as predictive cues in HRI-an eye-tracking study.

Robots currently provide only a limited amount of information about their future movements to human collaborators. In human interaction, communication through gaze can be helpful by intuitively directing attention to specific targets. Whether and how this mechanism could benefit the interaction with robots and how a design of predictive robot eyes in general should look like is not well understood. In a between-subjects design, four different types of eyes were therefore compared with regard to their attention directing potential: a pair of arrows, human eyes, and two anthropomorphic robot eye designs. For this purpose, 39 subjects performed a novel, screen-based gaze cueing task in the laboratory. Participants' attention was measured using manual responses and eye-tracking. Information on the perception of the tested cues was provided through additional subjective measures. All eye models were overall easy to read and were able to direct participants' attention. The anthropomorphic robot eyes were most efficient at shifting participants' attention which was revealed by faster manual and saccadic reaction times. In addition, a robot equipped with anthropomorphic eyes was perceived as being more competent. Abstract anthropomorphic robot eyes therefore seem to trigger a reflexive reallocation of attention. This points to a social and automatic processing of such artificial stimuli.

The robot eyes don't have it. The presence of eyes on collaborative robots yields marginally higher user trust but lower performance

Eye gaze is a prominent feature of human social lives, but little is known on whether fitting eyes on machines makes humans trust them more. In this study we compared subjective and objective markers of human trust when collaborating with eyed and non-eyed robots of the same type. We used virtual reality scenes in which we manipulated distance and the presence of eyes on a robot's display during simple collaboration scenes. We found that while collaboration with eyed cobots resulted in slightly higher subjective trust ratings, the objective markers such as pupil size and task completion time indicated it was in fact less comfortable to collaborate with eyed robots. These findings are in line with recent suggestions that anthropomorphism may be actually a detrimental feature of collaborative robots. These findings also show the complex relationship between human objective and subjective markers of trust when collaborating with artificial agents.

Home Surveillance Using Robotic Eye

Abstract: Security is the primary concern for everyone. This project is aimed at providing security to homes, banks, museum using robotic eye surveillance system. This project is used to detect the motion using sound technology and provides live streaming when required. Here, a technique of robotic eye surveillance is presented that uses active/passive sound motions to capture and record intruders. This project comprises an eye that mimics a robot eye and a servo motor controlled by Arduino NANO that allow the robotic eye to move. There is of course a sound level measurement and its visualization through a mounted camera inside the eye which is ESP32-CAM, which now acts as an input. And it provides surveillance in 3 directions x, y and z. So, how can the robot know the time it should react? And for what signals it should react? It is decided by time of sounds. The difference is done with the frequencies that are involved and sudden increase of sound intensity. Since 2 microphones are used for direction of sound detection and the continuous sound measurement shows the stepping up of the sound and eventually captures.

Integrated design and prototyping of a robotic eye system for ocular and craniofacial trauma simulators

This article presents the development of a prototype robotic eye-motion system for a novel simulator of ocular and craniofacial trauma that was developed for practical skills training of eye surgeons and first responders. The simulator fills a gap in the ophthalmological training domain, specifically between virtual reality-based systems for cataract and retinal repair and part-task trainers without quantitative measurement capabilities. Combining physical anatomical modules, instrument tracking and embedded sensors with a data acquisition/feedback system in a portable bench-top structure, it offers an alternative to animal- and cadaver-based training. The prototype robotic eye system described includes multiple human eye globe motion features: eye pitch and yaw motion, proptosis, and sensing of applied force to detect pressure/load applied to the globe.

Review on challenges for robotic eye surgery; surgical systems, technologies, cost-effectiveness, and controllers.

In recent decades, a number of surgical systems have been developed and are applied for a growing variety of surgeries. This review will consider the significant challenges of robotic surgery for the eye. These challenges take into account the different eye diseases, available technologies, and costs in different surgical systems for the eye. The conditions of a suitable controller will be discussed with consideration of relevant control engineering concepts. Comparison is made between the different characteristics of surgical robots for the eye. In this review, some comparisons will be made in eye surgical robots, control algorithms, sensors in surgical robots, communication protocols, and actuators.