Robot Perception Research Articles

Hardware Technology of Vision-Based Tactile Sensor: A Review

A vision-based tactile sensor (VBTS) is an innovative optical sensor widely applied in robotic perception. The VBTS consists of a contact module, an illumination module, and a camera module. Its hardware technologies contain preparation, performance, functions, materials, and optimization design of components. However, the current literature lacks a review of hardware technologies to formulate a complete manufacturing scheme. Therefore, this article focuses on hardware technologies and reviewed the literature over the past five years. We analyze the core components of each module and sort out a technical route. The current challenges and problems of hardware technologies are discussed to propose some feasible solutions. Considering cross-disciplinary applications, we think that multidisciplinary hardware technologies are expected to promote the development of next-generation VBTSs. In addition, we aim to provide a valuable guideline for developers to improve the performance of VBTSs.

Electric-field-coupled oscillators for collective electrochemical perception in biohybrid robotics

This work explores the application of nonlinear oscillators coupled by an electric field in water, inspired by weakly electric fish. Such coupled oscillators operate in clear and colloidal (mud, bottom silt) water and represent a collective electrochemical sensor that is sensitive to global environmental parameters, the geometry of the common electric field and spatial dynamics of autonomous underwater vehicles (AUVs). Implemented in hardware and software, this approach can be used to create global awareness in a group of robots, which possess limited sensing and communication capabilities. Using oscillators from different AUVs enables extension of the range limitations related to the electric dipole of a single AUV. Applications of this technique are demonstrated for detecting the number of AUVs, distances between them, perception of dielectric objects and synchronization of behavior. Recognizing self-/nonself-generated signals by electric fish is re-embodied in a technological way through an ‘electrical mirror’ for discrimination between ‘collective self’ and ‘collective nonself’. These approaches have been implemented in several research projects with bioinspired/biohybrid systems in fresh and salt water, and electrochemical sensing in fluidic media.

What do Members of Parliament in India Think of Robots? Validation of the Frankenstein Syndrome Questionnaire and Comparison with Other Population Groups

India is the second largest country in the world in terms of population and thus a considerable market for potential future robot applications as well as a location for manufacturing and production. While work has started to explore attitudes towards robots, very little is known about the perceptions of robots in India, particularly of political leaders who have the ability to effect rapid change. The present study administered the 30-item Frankenstein Syndrome Questionnaire to 31 Lok Sabha (Lower House) and Rajya Sabah (Upper House) members of the Indian Parliament (MPs) as well as doctors (n = 94), medical students (n = 493), and engineering students (n = 1104) for comparative purposes. Because no information had been available about the psychometric properties of the scale for use in India, a prior Rasch analysis explored the suitability of the commonly used five-factor model. The five subscales did not possess sufficient reliability, and a more psychometrically robust 26-item two-factor model (positive and negative attitudes) was utilized instead. The results revealed a higher degree of positive attitudes in MPs and doctors as compared to the two student groups. Negative attitudes, on the other hand, were strongest in doctors, followed by students. MPs had significantly less negative views compared to all other comparison groups. This study provides valuable insights into attitudes towards robots in India. In general, MPs appear to have more favourable views than comparison groups in India. A slightly shorter and more parsimonious version of the Frankenstein Syndrome Questionnaire has now also been proposed, with improved psychometric properties.

Finding ikigai: How robots can support meaning in later life.

Previous research in human-robot interaction has explored using robots to increase objective and hedonic aspects of well-being and quality of life, but there is no literature on how robots might be used to support eudaimonic aspects of well-being (such as meaning in life). A sense of meaning has been shown to positively affect health and longevity. We frame our study around the Japanese concept of ikigai, which is widely used with Japanese older adults to enhance their everyday lives, and is closely related to the concept of eudaimonic well-being (EWB) known in Western countries. Using a mixed-methods and exploratory approach, including interviews with 17 older adults and the collection of 100 survey responses, we explored how older adults in the US experience a sense of meaning, and if and how a social robot could be used to help foster this sense. We find that meaning for older adults is often obtained by helping others, through family connections, and/or through activities of daily life, and that sources of meaning often differ based on the older adults' living situation. Assessing how meaning compares to happiness and social connection, we highlight general similarities and differences, and also find that living situation influences older adults' sources of happiness, desire for social connection, and barriers to well-being, in addition to companionship and happiness having a weaker correlation with meaning for those who live alone than for those who live with others. Additionally, we evaluated initial perceptions of a social robot (QT) meant to enhance ikigai and overall well-being, finding mostly positive perceptions, though those who live alone also reported being less willing to adopt a social robot into their homes. Using both data collected on older adults' meaning and the potential use of QT to support meaning, we make several design recommendations with regards to using robots to enhance ikigai, such as by prompting daily reflecting, enhancing family bonds, and suggesting new experiences and volunteer opportunities.

Battery-Free UHF-RFID Sensors-Based SLAM for In-Pipe Robot Perception

Water utilities across the globe are concerned with the inspection and replacement of buried metallic water pipes due to corrosion-related structural damages. Internal pipe linings are commonly used as a renewal method to improve structural strength as they are regarded to be a less expensive alternative to costly and time-consuming pipe replacements. However, linings are also prone to failure as well. Therefore, water authorities regularly monitor lining performance, where defect evolution over a long period of time is an important parameter to note. It requires an accurate in-pipe robot localization technology. In this article, we propose a novel method for in-pipe robot localization and tag mapping that uses battery-free ultra high frequency (UHF)-radio-frequency identification (RFID) sensor wireless signals. It utilizes a signal mapping approach in combination with a tailored pose-graph simultaneous localization and mapping (SLAM) algorithm. Evaluation results of a field-extracted pipe sample from Sydney Water’s distribution network show that the proposed approach is capable of localizing the robot within 2.5 cm accuracy in a 50-m equivalent pipe with an unknown UHF-RFID distribution. The proposed approach outperformed other reported similar work in the literature.

Wireless Flexible Magnetic Tactile Sensor with Super-Resolution in Large-Areas.

Tactile recognition is among the basic survival skills of human beings, and advances in tactile sensor technology have been adopted in various fields, bringing benefits such as outstanding performance in manipulating objects and general human-robot interactions. However, promoting enhanced perception of the existing tactile sensors is limited by their sensor array arrangement and wire-connected design. Here we present a wireless flexible magnetic tactile sensor (FMTS) consisting of a multidirection magnetized flexible film (perception module) and a contactless Hall sensor (signal receiving module). The flexible magnetic film is composed of NdFeB microparticles and soft silicone elastomer microparticles, and it transfers the unambiguous transduction of external force position and magnitude into magnetic signals. Benefiting from the specific magnetization arrangement and clustering algorithm, only one Hall sensor is needed in FMTS to perceive the magnitude and position of the contact spot simultaneously with super-resolution (2.1 mm average error) on a large area (3600 mm2), and the effective working distance is also greatly extended (∼30 mm), allowing for the full softness and adaptability to diverse conditions. We anticipate that this design will promote the development of soft tactile sensors and their integration into human-robot interaction and humanoid robot perception.

Gender detection in children’s speech utterances for human-robot interaction

The human voice speech essentially includes paralinguistic information used in many real-time applications. Detecting the children’s gender is considered a challenging task compared to the adult’s gender. In this study, a system for human-robot interaction (HRI) is proposed to detect the gender in children’s speech utterances without depending on the text. The robot's perception includes three phases: Feature’s extraction phase where four formants are measured at each glottal pulse and then a median is calculated across these measurements. After that, three types of features are measured which are formant average (AF), formant dispersion (DF), and formant position (PF). Feature’s standardization phase where the measured feature dimensions are standardized using the z-score method. The semantic understanding phase is where the children’s gender is detected accurately using the logistic regression classifier. At the same time, the action of the robot is specified via a speech response using the text to speech (TTS) technique. Experiments are conducted on the Carnegie Mellon University (CMU) Kids dataset to measure the suggested system’s performance. In the suggested system, the overall accuracy is 98%. The results show a relatively clear improvement in terms of accuracy of up to 13% compared to related works that utilized the CMU Kids dataset.

DelTact: A Vision-Based Tactile Sensor Using a Dense Color Pattern

Tactile sensing is an essential perception for robots to complete dexterous tasks. As a promising tactile sensing technique, vision-based tactile sensors have been developed to improve robot performance in manipulation and grasping. Here we propose a new design of a vision-based tactile sensor, DelTact. The sensor uses a modular hardware architecture for compactness whilst maintaining a contact measurement of full resolution ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$798\times 586$</tex-math></inline-formula> ) and large area (675 mm <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{2}$</tex-math></inline-formula> ). Moreover, it adopts an improved dense random color pattern based on the previous version to achieve high accuracy of contact deformation tracking. In particular, we optimize the color pattern generation process and select the appropriate pattern for coordinating with a dense optical flow algorithm under a real-world experimental sensory setting. The optical flow obtained from the raw image is processed to determine shape and force distribution on the contact surface. We also demonstrate the method to extract contact shape and force distribution from the raw images. Experimental results demonstrate that the sensor is capable of providing tactile measurements with low error and high frequency (40 Hz).

Social Attitude Towards a Robot is Promoted by Motor-Induced Embodiment Independently of Spatial Perspective

Humans can enter into social relations with robots through different spatial perspectives. In the telepresence perspective, the human sees through the eyes of the robot. In the face-to-face perspective the human faces the robot and interacts in a self-other relation. During robotic telepresence, we have shown that embodiment into robots can be promoted by reciprocal and synchronous stimuli in the form of intentional movements or passive tactile stimulations. Here we investigate the impact of different spatial perspectives coupled to these sensory-motor manipulations on human subjects’ social perception of robots. Through a series of experiments, we demonstrate that independent of the telepresence vs. face-to-face self-perspective, the sense of agency as induced by synchronous human-robot movements is crucial for generating positive changes of robot acceptability. We suggest that motor intentionality most clearly influences our social perception of robots.

View Planning for Object Pose Estimation Using Point Clouds: An Active Robot Perception Approach

This letter considers the object pose estimation problem for robotic tasks where the robot end-effector is mounted with a vision sensor to collect point clouds from multiple views. The focus of this work is on generating and planning sensor views online based on the feedback from point cloud analysis to improve point cloud quality and quantity for pose estimation. To this end, we propose a novel active robot perception framework to generate sensor views online based on the collected data to improve the estimated pose. This framework includes the following components: evaluating the quality and quantity of the point clouds, finding a minimum number of optimal sensor views, and calculating valid robot poses corresponding to the sensor views. Tools and techniques from combinatorial optimization, mixed-integer programming, and constrained nonlinear optimization are utilized to formulate and solve the sensor view generation and robot pose problems. Extensive experiments were conducted to evaluate the proposed framework, and a representative sample of the results are provided. For baseline comparison, the proposed method is compared with two view planning methods from reconstruction, and the results show the benefits of incorporating the proposed active feedback mechanism for pose estimation. The overall effectiveness of the active perception strategy is shown through improved pose estimation accuracy.

Closed-Form Error Propagation on $SE_{n}(3)$ Group for Invariant EKF With Applications to VINS

Pose estimation is important for robotic perception, path planning, etc. Robot poses can be modeled on matrix Lie groups and are usually estimated via filter-based methods. In this letter, we establish the closed-form formula for the error propagation for the Invariant extended Kalman filter (IEKF) in the presence of random noises and apply it to vision-aided inertial navigation. Moreover, we use the theoretic results to add the compensation parts for IEKF. We evaluate our algorithms via numerical simulations and experiments on the OPENVINS platform. Both simulations and the experiments performed on the public EuRoC MAV datasets demonstrate that our algorithm in particular parameters settings outperforms some state-of-art filter-based methods such as the quaternion-based EKF, first estimates Jacobian EKF, etc. The techniques of choice on the parameters are worth further investigating.

Vision-Based Sensing for Electrically-Driven Soft Actuators

Developing reliable control strategies in soft robotics requires advances in soft robot perception. However, current soft robotic sensors pose many performance limitations, and available materials and manufacturing techniques complicate soft sensorized robot design. To address these long-standing needs, we introduce a method for using vision to sensorize robust, electrically-driven soft robotic actuators constructed from a new class of architected materials. Specifically, we use cameras positioned within the hollow interiors of handed shearing auxetic (HSA) actuators to record deformation during motion. We train a convolutional neural network (CNN) that maps the visual feedback to the actuator's tip pose. Our model provides predictions with sub-millimeter accuracy from only six minutes of training data, while remaining lightweight with an inference time of 18 milliseconds per frame. We also develop a model that additionally predicts the horizontal tip force acting on the actuator and generalizes to previously unseen forces. Finally, we demonstrate the viability of our sensorization strategy for contact-rich applications by training a CNN that predicts the tip pose accurately during tactile interactions. Overall, our methods present a reliable vision-based approach for designing sensorized soft robots built from electrically-actuated, architected materials.

EV-Catcher: High-Speed Object Catching Using Low-Latency Event-Based Neural Networks

Event-based sensors have recently drawn increasing interest in robotic perception due to their lower latency, higher dynamic range, and lower bandwidth requirements compared to standard CMOS-based imagers. These properties make them ideal tools for real-time perception tasks in highly dynamic environments. In this work, we demonstrate an application where event cameras excel: accurately estimating the impact location of fast-moving objects. We introduce a lightweight event representation called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Binary Event History Image</i> (BEHI) to encode event data at low latency, as well as a learning-based approach that allows real-time inference of a confidence-enabled control signal to the robot. To validate our approach, we present an experimental catching system in which we catch fast-flying ping-pong balls. We show that the system is capable of achieving a success rate of 81% in catching balls targeted at different locations, with a velocity of up to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$13\,\text{m/s}$</tex-math></inline-formula> even on compute-constrained embedded platforms such as the Nvidia Jetson NX.

Discrete-Continuous Smoothing and Mapping

We describe a general approach for maximum <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">a posteriori</i> (MAP) inference in a class of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">discrete-continuous factor graphs</i> commonly encountered in robotics applications. While there are openly available tools providing flexible and easy-to-use interfaces for specifying and solving inference problems formulated in terms of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">either</i> discrete <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">or</i> continuous graphical models, at present, no similarly general tools exist enabling the same functionality for <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">hybrid</i> discrete-continuous problems. We aim to address this problem. In particular, we provide a library, DC-SAM, extending existing tools for inference problems defined in terms of factor graphs to the setting of discrete-continuous models. A key contribution of our work is a novel solver for efficiently recovering approximate solutions to discrete-continuous inference problems. The key insight to our approach is that while joint inference over continuous and discrete state spaces is often hard, many commonly encountered discrete-continuous problems can naturally be split into a “discrete part” and a “continuous part” that can individually be solved easily. Leveraging this structure, we optimize discrete and continuous variables in an alternating fashion. In consequence, our proposed work enables straightforward representation of and approximate inference in discrete-continuous graphical models. We also provide a method to approximate the uncertainty in estimates of both discrete and continuous variables. We demonstrate the versatility of our approach through its application to distinct robot perception applications, including robust pose graph optimization, and object-based mapping and localization.

Deep leaning-based ultra-fast stair detection

Staircases are some of the most common building structures in urban environments. Stair detection is an important task for various applications, including the environmental perception of exoskeleton robots, humanoid robots, and rescue robots and the navigation of visually impaired people. Most existing stair detection algorithms have difficulty dealing with the diversity of stair structure materials, extreme light and serious occlusion. Inspired by human perception, we propose an end-to-end method based on deep learning. Specifically, we treat the process of stair line detection as a multitask involving coarse-grained semantic segmentation and object detection. The input images are divided into cells, and a simple neural network is used to judge whether each cell contains stair lines. For cells containing stair lines, the locations of the stair lines relative to each cell are regressed. Extensive experiments on our dataset show that our method can achieve 81.49% accuracy, 81.91% recall and 12.48 ms runtime, and our method has higher performance in terms of both speed and accuracy than previous methods. A lightweight version can even achieve 300+ frames per second with the same resolution.

Deep Q-network for social robotics using emotional social signals.

Social robotics represents a branch of human-robot interaction dedicated to developing systems to control the robots to operate in unstructured environments with the presence of human beings. Social robots must interact with human beings by understanding social signals and responding appropriately to them. Most social robots are still pre-programmed, not having great ability to learn and respond with actions adequate during an interaction with humans. Recently more elaborate methods use body movements, gaze direction, and body language. However, these methods generally neglect vital signs present during an interaction, such as the human emotional state. In this article, we address the problem of developing a system to turn a robot able to decide, autonomously, what behaviors to emit in the function of the human emotional state. From one side, the use of Reinforcement Learning (RL) represents a way for social robots to learn advanced models of social cognition, following a self-learning paradigm, using characteristics automatically extracted from high-dimensional sensory information. On the other side, Deep Learning (DL) models can help the robots to capture information from the environment, abstracting complex patterns from the visual information. The combination of these two techniques is known as Deep Reinforcement Learning (DRL). The purpose of this work is the development of a DRL system to promote a natural and socially acceptable interaction among humans and robots. For this, we propose an architecture, Social Robotics Deep Q-Network (SocialDQN), for teaching social robots to behave and interact appropriately with humans based on social signals, especially on human emotional states. This constitutes a relevant contribution for the area since the social signals must not only be recognized by the robot but help him to take action appropriated according to the situation presented. Characteristics extracted from people’s faces are considered for extracting the human emotional state aiming to improve the robot perception. The development and validation of the system are carried out with the support of SimDRLSR simulator. Results obtained through several tests demonstrate that the system learned satisfactorily to maximize the rewards, and consequently, the robot behaves in a socially acceptable way.

Progress in symmetry preserving robot perception and control through geometry and learning.

This article reports on recent progress in robot perception and control methods developed by taking the symmetry of the problem into account. Inspired by existing mathematical tools for studying the symmetry structures of geometric spaces, geometric sensor registration, state estimator, and control methods provide indispensable insights into the problem formulations and generalization of robotics algorithms to challenging unknown environments. When combined with computational methods for learning hard-to-measure quantities, symmetry-preserving methods unleash tremendous performance. The article supports this claim by showcasing experimental results of robot perception, state estimation, and control in real-world scenarios.

Dances with Social Robots: A Pilot Study at Long-Term Care

Dance therapy can have significant physical, emotional and cognitive benefits for older adults. In particular, social robots can be developed to autonomously facilitate dance sessions to engage these individuals with the aim of improving quality of life. To successfully integrate and promote long-term use of social robots into long-term care homes for such recreational activities, it is important to explore both residents’ and staff’s perceptions of such robots. In this paper, we present the first pilot human–robot interaction study that investigates the overall experiences and attitudes of both residents and staff in a long-term care home for robot-facilitated dance sessions. In general, the questionnaire results from our study showed that both staff and residents had positive attitudes towards the robot-facilitated dance activity. Encouraging trends showed residents had higher ratings for statements on perceived ease of use, safety, and enjoyment than the staff. However, the staff had a statistically significantly higher rating for willingness to use the robots for dance facilitation. Some key statistical differences were also determined with respect to: (1) gender within the resident group (men had higher ratings for the robots being useful in helping facilitate recreational activities), as well as between staff and residents (resident men had higher perceived safety), and (2) prior robot experience (residents with limited prior experience had higher ratings on perceived ease of use and perceived enjoyment than staff with the same level of experience). The robot-facilitated dance activity was positively received by both older adults and staff as an activity of daily living that can enhance wellbeing while also being safe, easy to use and enjoyable.

Is It Personal? The Impact of Personally Relevant Robotic Failures (PeRFs) on Humans' Trust, Likeability, and Willingness to Use the Robot.

In three laboratory experiments, we examine the impact of personally relevant failures (PeRFs) on users’ perceptions of a collaborative robot. PeR is determined by how much a specific issue applies to a particular person, i.e., it affects one's own goals and values. We hypothesized that PeRFs would reduce trust in the robot and the robot's Likeability and Willingness to Use (LWtU) more than failures that are not personal to participants. To achieve PeR in human–robot interaction, we utilized three different manipulation mechanisms: (A) damage to property, (B) financial loss, and (C) first-person versus third-person failure scenarios. In total, 132 participants engaged with a robot in person during a collaborative task of laundry sorting. All three experiments took place in the same experimental environment, carefully designed to simulate a realistic laundry sorting scenario. Results indicate that the impact of PeRFs on perceptions of the robot varied across the studies. In experiments A and B, the encounters with PeRFs reduced trust significantly relative to a no failure session. But not entirely for LWtU. In experiment C, the PeR manipulation had no impact. The work highlights challenges and adjustments needed for studying robotic failures in laboratory settings. We show that PeR manipulations affect how users perceive a failing robot. The results bring about new questions regarding failure types and their perceived severity on users' perception of the robot. Putting PeR aside, we observed differences in the way users perceive interaction failures compared (experiment C) to how they perceive technical ones (A and B).

Employees’ perception of robots and robot-induced unemployment in hospitality industry under COVID-19 pandemic

The impact of the pandemic is driving the recent upsurge in service automation and the adoption of service robots in the hospitality industry. As service paradigm and customer expectations shift from conventional customized and personalized services towards a digitalized service environment, such customer orientation may favor using service robots at scales that could render service employees redundant. This study aims to answer the above question by investigating service employees’ perceptions of service robots. Data solicited from 405 service employees in the United States of America via Amazon’s MTurk were analyzed using structural equation modeling. The result revealed that employees’ awareness of adopting and using service robots significantly impacts their perception of robot-induced unemployment. Further, results indicated that the perception of robots’ social skills significantly influences service employees’ perception of robot-induced unemployment. Employee status was found to moderate the relationships mentioned above. Specifically, entry-level employees perceive the unemployment risk more than managers.