Robot Perception Research Articles

Unleashing mixed-reality capability in Deep Reinforcement Learning-based robot motion generation towards safe human–robot collaboration

The integration of human–robot collaboration yields substantial benefits, particularly in terms of enhancing flexibility and efficiency within a range of mass-personalized manufacturing tasks, for example, small-batch customized product inspection and assembly/disassembly. Meanwhile, as human–robot collaboration lands broader in manufacturing, the unstructured scene and operator uncertainties are increasingly involved and considered. Consequently, it becomes imperative for robots to execute in a safe and adaptive manner rather than solely relying on pre-programmed instructions. To tackle it, a systematic solution for safe robot motion generation in human–robot collaborative activities is proposed, leveraging mixed-reality technologies and Deep Reinforcement Learning. This solution covers the entire process of collaboration starting with an intuitive interface that facilitates bare-hand task command transmission and scene coordinate transformation before the collaboration begins. In particular, mixed-reality devices are employed as effective tools for representing the state of humans, robots, and scenes. This enables the learning of an end-to-end Deep Reinforcement Learning policy that addresses both the uncertainties in robot perception and decision-making in an integrated manner. The proposed solution also implements policy simulation-to-reality deployment, along with motion preview and collision detection mechanisms, to ensure safe robot motion execution. It is hoped that this work could inspire further research in human–robot collaboration to unleash and exploit the powerful capabilities of mixed reality.

Active Dual Line-Laser Scanning for Depth Imaging of Piled Agricultural Commodities for Itemized Processing Lines.

The accurate depth imaging of piled products provides essential perception for the automated selection of individual objects that require itemized food processing, such as fish, crabs, or fruit. Traditional depth imaging techniques, such as Time-of-Flight and stereoscopy, lack the necessary depth resolution for imaging small items, such as food commodities. Although structured light methods such as laser triangulation have high depth resolution, they depend on conveyor motion for depth scanning. This manuscript introduces an active dual line-laser scanning system for depth imaging static piled items, such as a pile of crabs on a table, eliminating the need for conveyor motion to generate high-resolution 3D images. This advancement benefits robotic perception for loading individual items from a pile for itemized food processing. Leveraging a unique geometrical configuration and laser redundancy, the dual-laser strategy overcomes occlusions while reconstructing a large field of view (FOV) from a long working distance. We achieved a depth reconstruction MSE of 0.3 mm and an STD of 0.5 mm on a symmetrical pyramid stage. The proposed system demonstrates that laser scanners can produce depth maps of complex items, such as piled Chesapeake Blue Crab and White Button mushrooms. This technology enables 3D perception for automated processing lines and offers broad applicability for quality inspection, sorting, and handling of piled products.

Enhancing Robotic Perception through Synchronized Simulation and Physical Common-Sense Reasoning.

We introduce both conceptual and empirical findings arising from the amalgamation of a robotics cognitive architecture with an embedded physics simulator, aligning with the principles outlined in the intuitive physics literature. The employed robotic cognitive architecture, named CORTEX, leverages a highly efficient distributed working memory known as deep state representation. This working memory inherently encompasses a fundamental ontology, state persistency, geometric and logical relationships among elements, and tools for reading, updating, and reasoning about its contents. Our primary objective is to investigate the hypothesis that the integration of a physics simulator into the architecture streamlines the implementation of various functionalities that would otherwise necessitate extensive coding and debugging efforts. Furthermore, we categorize these enhanced functionalities into broad types based on the nature of the problems they address. These include addressing challenges related to occlusion, model-based perception, self-calibration, scene structural stability, and human activity interpretation. To demonstrate the outcomes of our experiments, we employ CoppeliaSim as the embedded simulator and both a Kinova Gen3 robotic arm and the Open-Manipulator-P as the real-world scenarios. Synchronization is maintained between the simulator and the stream of real events. Depending on the ongoing task, numerous queries are computed, and the results are projected into the working memory. Participating agents can then leverage this information to enhance overall performance.

Point-to-Spike Residual Learning for Energy-Efficient 3D Point Cloud Classification

Spiking neural networks (SNNs) have revolutionized neural learning and are making remarkable strides in image analysis and robot control tasks with ultra-low power consumption advantages. Inspired by this success, we investigate the application of spiking neural networks to 3D point cloud processing. We present a point-to-spike residual learning network for point cloud classification, which operates on points with binary spikes rather than floating-point numbers. Specifically, we first design a spatial-aware kernel point spiking neuron to relate spiking generation to point position in 3D space. On this basis, we then design a 3D spiking residual block for effective feature learning based on spike sequences. By stacking the 3D spiking residual blocks, we build the point-to-spike residual classification network, which achieves low computation cost and low accuracy loss on two benchmark datasets, ModelNet40 and ScanObjectNN. Moreover, the classifier strikes a good balance between classification accuracy and biological characteristics, allowing us to explore the deployment of 3D processing to neuromorphic chips for developing energy-efficient 3D robotic perception systems.

Freedom comes at a cost?: An exploratory study on affordances' impact on users' perception of a social robot.

Along with the development of speech and language technologies, the market for speech-enabled human-robot interactions (HRI) has grown in recent years. However, it is found that people feel their conversational interactions with such robots are far from satisfactory. One of the reasons is the habitability gap, where the usability of a speech-enabled agent drops when its flexibility increases. For social robots, such flexibility is reflected in the diverse choice of robots' appearances, sounds and behaviours, which shape a robot's 'affordance'. Whilst designers or users have enjoyed the freedom of constructing a social robot by integrating off-the-shelf technologies, such freedom comes at a potential cost: the users' perceptions and satisfaction. Designing appropriate affordances is essential for the quality of HRI. It is hypothesised that a social robot with aligned affordances could create an appropriate perception of the robot and increase users' satisfaction when speaking with it. Given that previous studies of affordance alignment mainly focus on one interface's characteristics and face-voice match, we aim to deepen our understanding of affordance alignment with a robot's behaviours and use cases. In particular, we investigate how a robot's affordances affect users' perceptions in different types of use cases. For this purpose, we conducted an exploratory experiment that included three different affordance settings (adult-like, child-like, and robot-like) and three use cases (informative, emotional, and hybrid). Participants were invited to talk to social robots in person. A mixed-methods approach was employed for quantitative and qualitative analysis of 156 interaction samples. The results show that static affordance (face and voice) has a statistically significant effect on the perceived warmth of the first impression; use cases affect people's perceptions more on perceived competence and warmth before and after interactions. In addition, it shows the importance of aligning static affordance with behavioural affordance. General design principles of behavioural affordances are proposed. We anticipate that our empirical evidence will provide a clearer guideline for speech-enabled social robots' affordance design. It will be a starting point for more sophisticated design guidelines. For example, personalised affordance design for individual or group users in different contexts.

The Robot Rights and Responsibilities Scale: Development and Validation of a Metric for Understanding Perceptions of Robots’ Rights and Responsibilities

The discussion and debates surrounding the robot rights topic demonstrate vast differences in the possible philosophical, ethical, and legal approaches to this question. Without top-down guidance of mutually agreed upon legal and moral imperatives, the public’s attitudes should be an important component of the discussion. However, few studies have been conducted on how the general population views aspects of robot rights. The aim of the current study is to provide a new measurement that may facilitate such research. A Robot Rights and Responsibilities (RRR) scale is developed and tested. An exploratory factor analysis reveals a multi-dimensional construct with three factors—robots’ rights, responsibilities, and capabilities—which are found to concur with theoretically relevant metrics. The RRR scale is contextualized in the ongoing discourse about the legal and moral standing of non-human and artificial entities. Implications for people’s ontological perceptions of machines and suggestions for future empirical research are considered.

CASPER: Cognitive Architecture for Social Perception and Engagement in Robots

AbstractOur world is being increasingly pervaded by intelligent robots with varying degrees of autonomy. To seamlessly integrate themselves in our society, these machines should possess the ability to navigate the complexities of our daily routines even in the absence of a human’s direct input. In other words, we want these robots to understand the intentions of their partners with the purpose of predicting the best way to help them. In this paper, we present the initial iteration of cognitive architecture for social perception and engagement in robots: a symbolic cognitive architecture that uses qualitative spatial reasoning to anticipate the pursued goal of another agent and to calculate the best collaborative behavior. This is performed through an ensemble of parallel processes that model a low-level action recognition and a high-level goal understanding, both of which are formally verified. We have tested this architecture in a simulated kitchen environment and the results we have collected show that the robot is able to both recognize an ongoing goal and to properly collaborate towards its achievement. This demonstrates a new use of qualitative spatial relations applied to the problem of intention reading in the domain of human–robot interaction.

YOLOv8-PoseBoost: Advancements in Multimodal Robot Pose Keypoint Detection

In the field of multimodal robotics, achieving comprehensive and accurate perception of the surrounding environment is a highly sought-after objective. However, current methods still have limitations in motion keypoint detection, especially in scenarios involving small target detection and complex scenes. To address these challenges, we propose an innovative approach known as YOLOv8-PoseBoost. This method introduces the Channel Attention Module (CBAM) to enhance the network’s focus on small targets, thereby increasing sensitivity to small target individuals. Additionally, we employ multiple scale detection heads, enabling the algorithm to comprehensively detect individuals of varying sizes in images. The incorporation of cross-level connectivity channels further enhances the fusion of features between shallow and deep networks, reducing the rate of missed detections for small target individuals. We also introduce a Scale Invariant Intersection over Union (SIoU) redefined bounding box regression localization loss function, which accelerates model training convergence and improves detection accuracy. Through a series of experiments, we validate YOLOv8-PoseBoost’s outstanding performance in motion keypoint detection for small targets and complex scenes. This innovative approach provides an effective solution for enhancing the perception and execution capabilities of multimodal robots. It has the potential to drive the development of multimodal robots across various application domains, holding both theoretical and practical significance.

Human-Robot handover task intention recognition framework by fusing human digital twin and deep domain adaptation

Industry 5.0 is conceived to achieve the expected high level of smart manufacturing and the more proactive and adaptive human-robot collaboration (HRC). In particular, human-robot handover tasks (HRHTs) are a notably crucial aspect of HRC, and resolving how to motivate robots to comprehend human handover intentions constitutes an exigent issue that demands resolution. To address these problems, A framework that integrates hierarchical human digital twin (HHDT) and deep domain adaptation is proposed, where HHDT model is built to digitise the human physical and physiological state and enhance the robot's perception of humans, a feature extractor-based on bidirectional long-short-term memory and spatio-temporal graph convolution network (BiLSTM-ST-GCN) is devised to extract latent spatio-temporal features of HRHTs intentions. Additionally, the framework incorporates a deep domain adaptation layer (DDAL) that facilitates knowledge transfer of HRHTs intentions by accounting for the distribution differences between the source and target operator objects. Extensive experiments demonstrate the superior performance of the proposed framework for HRHTs intention recognition and extraction of domain invariant features.

Social perception of robots is shaped by beliefs about their minds

Roboticists often imbue robots with human-like physical features to increase the likelihood that they are afforded benefits known to be associated with anthropomorphism. Similarly, deepfakes often employ computer-generated human faces to attempt to create convincing simulacra of actual humans. In the present work, we investigate whether perceivers’ higher-order beliefs about faces (i.e., whether they represent actual people or android robots) modulate the extent to which perceivers deploy face-typical processing for social stimuli. Past work has shown that perceivers’ recognition performance is more impacted by the inversion of faces than objects, thus highlighting that faces are processed holistically (i.e., as Gestalt), whereas objects engage feature-based processing. Here, we use an inversion task to examine whether face-typical processing is attenuated when actual human faces are labeled as non-human (i.e., android robot). This allows us to employ a task shown to be differentially sensitive to social (i.e., faces) and non-social (i.e., objects) stimuli while also randomly assigning face stimuli to seem real or fake. The results show smaller inversion effects when face stimuli were believed to represent android robots compared to when they were believed to represent humans. This suggests that robots strongly resembling humans may still fail to be perceived as “social” due pre-existing beliefs about their mechanistic nature. Theoretical and practical implications of this research are discussed.

Systematic Review of Retinal Blood Vessels Segmentation Based on AI-driven Technique.

Image segmentation is a crucial task in computer vision and image processing, with numerous segmentation algorithms being found in the literature. It has important applications in scene understanding, medical image analysis, robotic perception, video surveillance, augmented reality, image compression, among others. In light of this, the widespread popularity of deep learning (DL) and machine learning has inspired the creation of fresh methods for segmenting images using DL and ML models respectively. We offer a thorough analysis of this recent literature, encompassing the range of ground-breaking initiatives in semantic and instance segmentation, including convolutional pixel-labeling networks, encoder-decoder architectures, multi-scale and pyramid-based methods, recurrent networks, visual attention models, and generative models in adversarial settings. We study the connections, benefits, and importance of various DL- and ML-based segmentation models; look at the most popular datasets; and evaluate results in this Literature.

Factors associated with intention to use care robots among people with physical disabilities

As the disabled population ages and the demand for care increases, Socially, the need for care robots is emerging but, perceptions of care robots among care recipients is unknown. To determine the level of intention to use care robots among care recipients and identify predictors of intention to use care robots. A cross-sectional survey was conducted using a convenience sample of 163 persons with disabilities from January to March 2022 at the Veterans Health Service Medical Center. Overall, 64.42% of respondents intended to use care robots. Predictors included perceived behavioral control, participants' perceptions of the caregiver's burden, attitude toward robot use, subjective norms, and age. These findings suggest that individuals who are community-dwelling desire the use care robots to maintain their independence and may provide useful insignt for the introduction various care robots in acute care and long-term care settings.

Autonomous Driving System Driven by Artificial Intelligence Perception Fusion

Perception, as the information input module of the automatic driving system, determines the lower limit of the entire automatic driving system. Both autonomous driving perception and robot perception are constantly approaching the real physical world through digital methods, and this real physical world is only limited to the scope of human perception, such as lane lines, traffic lights, driving obstacles, and so on. The main premise of this process is that humans already know the categories or properties of the physical world, and only allow machines and systems to replicate human responses. Whether it is a pure visual route or a multi-source fusion route, the essence is the difference between the perceptual system schemes, one focusing on the vertical and the other on the horizontal. The pure vision solution represented by Tesla or the multi-source sensor fusion file represented by Waymo. In fact, the perception module of the automatic driving system usually has multiple sensors to achieve information redundancy and information complementarity through multiple dimensions, but there is the possibility of information conflict between different sensors. This paper aims at the advantages of perception-driven artificial intelligence to achieve breakthroughs in autonomous driving innovation, and analyzes how perception fusion drive is applied to the practical application of autonomous driving, so as to analyze the future development prospects of artificial intelligence.

Computational design of ultra-robust strain sensors for soft robot perception and autonomy

Compliant strain sensors are crucial for soft robots’ perception and autonomy. However, their deformable bodies and dynamic actuation pose challenges in predictive sensor manufacturing and long-term robustness. This necessitates accurate sensor modelling and well-controlled sensor structural changes under strain. Here, we present a computational sensor design featuring a programmed crack array within micro-crumples strategy. By controlling the user-defined structure, the sensing performance becomes highly tunable and can be accurately modelled by physical models. Moreover, they maintain robust responsiveness under various demanding conditions including noise interruptions (50% strain), intermittent cyclic loadings (100,000 cycles), and dynamic frequencies (0–23 Hz), satisfying soft robots of diverse scaling from macro to micro. Finally, machine intelligence is applied to a sensor-integrated origami robot, enabling robotic trajectory prediction (<4% error) and topographical altitude awareness (<10% error). This strategy holds promise for advancing soft robotic capabilities in exploration, rescue operations, and swarming behaviors in complex environments.

Paraffin-Enabled Superlattice Customization for a Photostimulated Gradient-Responsive Artificial Reflex Arc.

The development of photostimulated-motion artificial reflex arcs - a neural circuit inspired by light-driven motion reflexes - holds significant promises for advancements in robotic perception, navigation, and motion control. However, the fabrication of such systems, especially those that accommodate multiple actions and exhibit gradient responses, remains challenging. Here, a gradient-responsive photostimulated-motion artificial reflex arc is developed by integrating a programmable and tunable photoreceptor based on folded MoS2 at different twist angles. The twisted folded bilayer MoS2 used as photoreceptors can be customized via the transfer technique using patternable paraffin, where the twist angle and fold-line could be controlled. The photoluminescence (PL) intensity is 3.7 times higher at a twist angle of 29° compared to that at 0°, showing a monotonically decreasing indirect bandgap. Through tunable interlayer carrier transport, photoreceptors fabricated using folded bilayer MoS2 at different twist angles demonstrate gradient response time, enabling the photostimulated-motion artificial reflex arc for multiaction responses. They are transformed to digital command flow and studied via machine learning to control the gestures of a robotic hand, showing a prototype of photostimulated gradient-responsive artificial reflex arcs for the first time. This work provides a unique idea for developing intelligent soft robots and next-generation human-computer interfaces.

Cognitive Enhancement of Robot Path Planning and Environmental Perception Based on Gmapping Algorithm Optimization

In the logistics warehouse environment, the autonomous navigation and environment perception of the logistics sorting robot are two key challenges. To deal with the complex obstacles and cargo layout in a warehouse, this study focuses on improving the robot perception and navigation system to achieve efficient path planning and safe motion control. For this purpose, a scheme based on an improved Gmapping algorithm is proposed to construct a high-precision map inside a warehouse through the efficient scanning and processing of environmental data by robots. While the improved algorithm effectively integrates sensor data with robot position information to realize the real-time modeling and analysis of warehouse environments. Consequently, the precise mapping results provide a reliable navigation basis for the robot, enabling it to make intelligent path planning and obstacle avoidance decisions in unknown or dynamic environments. The experimental results show that the robot using the improved Gmapping algorithm has high accuracy and robustness in identifying obstacles and an effectively reduced navigation error, thus improving the intelligence level and efficiency of logistics operations. The improved algorithm significantly enhances obstacle detection rates, increasing them by 4.05%. Simultaneously, it successfully reduces map size accuracy errors by 1.4% and angle accuracy errors by 0.5%. Additionally, the accuracy of the robot’s travel distance improves by 2.4%, and the mapping time is reduced by nine seconds. Significant progress has been made in achieving high-precision environmental perception and intelligent navigation, providing reliable technical support and solutions for autonomous operations in logistics warehouses.

Foundations of spatial perception for robotics: Hierarchical representations and real-time systems

3D spatial perception is the problem of building and maintaining an actionable and persistent representation of the environment in real-time using sensor data and prior knowledge. Despite the fast-paced progress in robot perception, most existing methods either build purely geometric maps (as in traditional SLAM) or “flat” metric-semantic maps that do not scale to large environments or large dictionaries of semantic labels. The first part of this paper is concerned with representations: we show that scalable representations for spatial perception need to be hierarchical in nature. Hierarchical representations are efficient to store, and lead to layered graphs with small treewidth, which enable provably efficient inference. We then introduce an example of hierarchical representation for indoor environments, namely a 3D scene graph, and discuss its structure and properties. The second part of the paper focuses on algorithms to incrementally construct a 3D scene graph as the robot explores the environment. Our algorithms combine 3D geometry (e.g., to cluster the free space into a graph of places), topology (to cluster the places into rooms), and geometric deep learning (e.g., to classify the type of rooms the robot is moving across). The third part of the paper focuses on algorithms to maintain and correct 3D scene graphs during long-term operation. We propose hierarchical descriptors for loop closure detection and describe how to correct a scene graph in response to loop closures, by solving a 3D scene graph optimization problem. We conclude the paper by combining the proposed perception algorithms into Hydra, a real-time spatial perception system that builds a 3D scene graph from visual-inertial data in real-time. We showcase Hydra’s performance in photo-realistic simulations and real data collected by a Clearpath Jackal robots and a Unitree A1 robot. We release an open-source implementation of Hydra at https://github.com/MIT-SPARK/Hydra.

Informal caregivers’ perception of assistive robots in eldercare

The acceptance of assistive robots in eldercare is an emerging field of research, particularly due to the longevity of the ageing population. This study aimed to advance the modelling of the value-based adoption model to assess assistive robots’ acceptance in eldercare. A group of 387 informal caregivers was surveyed. Perceived ease of use, financial risk, perceived usefulness and well-being were found to be the drivers of perceived value in the proposed model, whereas performance risk had no significant effect. Trust, well-being and perceived value were significant drivers of assistive robots’ acceptance by informal caregivers. The findings hold potential implications for policy and practice, suggesting specific measures that assistive robot firms and relevant parties could adopt. The findings offer feasible recommendations for assistive robot firms and other relevant parties to enhance trust, promote well-being and maximise perceived value in the design and implementation of assistive robot technologies. These practical measures can contribute to more successful adoption of assistive robots, benefiting informal caregivers, eldercare service providers and policymakers alike.

Self-administered questionnaires enhance emotion estimation of individuals with autism spectrum disorders in a robotic interview setting.

Robots offer many unique opportunities for helping individuals with autism spectrum disorders (ASD). Determining the optimal motion of robots when interacting with individuals with ASD is important for achieving more natural human-robot interactions and for exploiting the full potential of robotic interventions. Most prior studies have used supervised machine learning (ML) of user behavioral data to enable robot perception of affective states (i.e., arousal and valence) and engagement. It has previously been suggested that including personal demographic information in the identification of individuals with ASD is important for developing an automated system to perceive individual affective states and engagement. In this study, we hypothesized that assessing self-administered questionnaire data would contribute to the development of an automated estimation of the affective state and engagement when individuals with ASD are interviewed by an Android robot, which will be linked to implementing long-term interventions and maintaining the motivation of participants. Participants sat across a table from an android robot that played the role of the interviewer. Each participant underwent a mock job interview. Twenty-five participants with ASD (males 22, females 3, average chronological age = 22.8, average IQ = 94.04) completed the experiment. We collected multimodal data (i.e., audio, motion, gaze, and self-administered questionnaire data) to train a model to correctly classify the state of individuals with ASD when interviewed by an android robot. We demonstrated the technical feasibility of using ML to enable robot perception of affect and engagement of individuals with ASD based on multimodal data. For arousal and engagement, the area under the curve (AUC) values of the model estimates and expert coding were relatively high. Overall, the AUC values of arousal, valence, and engagement were improved by including self-administered questionnaire data in the classification. These findings support the hypothesis that assessing self-administered questionnaire data contributes to the development of an automated estimation of an individual's affective state and engagement. Given the efficacy of including self-administered questionnaire data, future studies should confirm the effectiveness of such long-term intervention with a robot to maintain participants' motivation based on the proposed method of emotion estimation.

Industrial design of 3D printing technology combined with assisted medical service robots

AbstractWith the continuous development and maturation of the era of intelligent manufacturing, there is a perpetual emergence of new information technology, control technology, and material technology, which are constantly accelerating 3D printing technology to advance to an unprecedented level. To achieve the safety perception interaction ability of auxiliary medical service robots, this study develops a direct write hybrid 3D printing auxiliary medical service robot system, enabling it to achieve temperature sensing function. Moreover, combined with linear interpolation algorithms, 3D printing technology has been improved to achieve improvement of system control accuracy. The results indicate that the apparent viscosity of the printing material Ag‐TPU is still greater than 2000 Pa s at a rate of 87 s−1. The change in resistance during 20% stretching is within 1.2 Ω, and the change is around 3 Ω during 30% stretching. When the preset temperature is 39.2°C, the absolute deviation is the smallest, about 0.03. When the preset temperature is 41.7°C, the maximum value is approximately 0.17. The absolute error of real‐time temperature collection for auxiliary medical service robots is less than 0.2°C at temperatures ranging from 38 to 42°C. Over the past 30 days of overall operation, the system has had 970 users, 3270 interactions, and 99.4% availability. This system improves the perception and interaction ability of auxiliary medical service robots, which has certain practical potential in the field of medical services.