Robot Perception Research Articles

General-Purpose Deep Learning Detection and Segmentation Models for Images from a Lidar-Based Camera Sensor

Over the last decade, robotic perception algorithms have significantly benefited from the rapid advances in deep learning (DL). Indeed, a significant amount of the autonomy stack of different commercial and research platforms relies on DL for situational awareness, especially vision sensors. This work explored the potential of general-purpose DL perception algorithms, specifically detection and segmentation neural networks, for processing image-like outputs of advanced lidar sensors. Rather than processing the three-dimensional point cloud data, this is, to the best of our knowledge, the first work to focus on low-resolution images with a 360° field of view obtained with lidar sensors by encoding either depth, reflectivity, or near-infrared light in the image pixels. We showed that with adequate preprocessing, general-purpose DL models can process these images, opening the door to their usage in environmental conditions where vision sensors present inherent limitations. We provided both a qualitative and quantitative analysis of the performance of a variety of neural network architectures. We believe that using DL models built for visual cameras offers significant advantages due to their much wider availability and maturity compared to point cloud-based perception.

Soft Robotic Perception System with Ultrasonic Auto-Positioning and Multimodal Sensory Intelligence

Flexible electronics such as tactile cognitive sensors have been broadly adopted in soft robotic manipulators to enable human-skin-mimetic perception. To achieve appropriate positioning for randomly distributed objects, an integrated guiding system is inevitable. Yet the conventional guiding system based on cameras or optical sensors exhibits limited environment adaptability, high data complexity, and low cost effectiveness. Herein, a soft robotic perception system with remote object positioning and a multimodal cognition capability is developed by integrating an ultrasonic sensor with flexible triboelectric sensors. The ultrasonic sensor is able to detect the object shape and distance by reflected ultrasound. Thereby the robotic manipulator can be positioned to an appropriate position to perform object grasping, during which the ultrasonic and triboelectric sensors can capture multimodal sensory information such as object top profile, size, shape, hardness, material, etc. These multimodal data are then fused for deep-learning analytics, leading to a highly enhanced accuracy in object identification (∼100%). The proposed perception system presents a facile, low-cost, and effective methodology to integrate positioning capability with multimodal cognitive intelligence in soft robotics, significantly expanding the functionalities and adaptabilities of current soft robotic systems in industrial, commercial, and consumer applications.

When Harry, the human, met Sally, the software robot: Metaphorical sensemaking and sensegiving around an emergent digital technology

Robotic process automation (RPA) is often used in organisational digitalisation efforts to automate work processes. RPA, and the software robots at its heart, is an equivocal and contentious technology. Adopting the products of theorising approach, this study views metaphors as central sensemaking and sensegiving devices that shape the interpretation of RPA among stakeholders towards a preferred reality of ways of seeing and experiencing software robots. The empirical materials are drawn from research in three Australasian organisations that have implemented RPA. Grounding our analysis in the domains-interaction model, we identified three root metaphors: person, robot and tool, their constitutive conceptual metaphors, and intended use as heuristics devices. Our findings show that metaphor is a powerful device that employees rely on to make sense of their experiences with a new digital technology that can potentially shape their roles, work practices and job design. In addition, managers and automation team members intentionally leverage metaphors to shape others’ perceptions of a software robot’s capabilities and limitations, its implication for human work and its expanding benefits for organisations over time, among others. Metaphor as a precursor to more formal theory provides scholars with a vocabulary to understand disparate experiences with an emergent automation technology that can be further developed to generate a theory of seeing automation and working with automated agents.

Ultrathin Mo2S3 Nanowire Network for High-Sensitivity Breathable Piezoresistive Electronic Skins.

Flexible piezosensing electronic skins (e-skins) have attracted considerable interest owing to their applications in real-time human-health monitoring, human-machine interactions, and soft bionic robot perception. However, the fabrication of piezosensing e-skins with high sensitivity, biological affinity, and good permeability at the same time is challenging. Herein, we designed and synthesized Mo2S3 nanowires by inserting ∞1[Mo2+S] chains between MoS2 interlayers. The resulting Mo2S3 nanowires feature high conductivity (4.9 × 104 S m-1) and a high aspect ratio (∼200). An ultrathin (∼500 nm) Mo2S3 nanowire network was fabricated using a simple liquid/liquid interface self-assembly method, showing high piezoresistive sensitivity (5.65 kPa-1), a considerably low pressure detection limit (0.08 Pa), and gratifying air permeability. Moreover, this nanowire network can be directly attached to human skin for real-time human pulse detection, finger movement monitoring, and sign language recognition, exhibiting excellent potential for health monitoring and human-machine interactions.

Lightweight and Energy-Aware Monocular Depth Estimation Models for IoT Embedded Devices: Challenges and Performances in Terrestrial and Underwater Scenarios.

The knowledge of environmental depth is essential in multiple robotics and computer vision tasks for both terrestrial and underwater scenarios. Moreover, the hardware on which this technology runs, generally IoT and embedded devices, are limited in terms of power consumption, and therefore, models with a low-energy footprint are required to be designed. Recent works aim at enabling depth perception using single RGB images on deep architectures, such as convolutional neural networks and vision transformers, which are generally unsuitable for real-time inferences on low-power embedded hardware. Moreover, such architectures are trained to estimate depth maps mainly on terrestrial scenarios due to the scarcity of underwater depth data. Purposely, we present two lightweight architectures based on optimized MobileNetV3 encoders and a specifically designed decoder to achieve fast inferences and accurate estimations over embedded devices, a feasibility study to predict depth maps over underwater scenarios, and an energy assessment to understand which is the effective energy consumption during the inference. Precisely, we propose the MobileNetV3S75 configuration to infer on the 32-bit ARM CPU and the MobileNetV3LMin for the 8-bit Edge TPU hardware. In underwater settings, the proposed design achieves comparable estimations with fast inference performances compared to state-of-the-art methods. Moreover, we statistically proved that the architecture of the models has an impact on the energy footprint in terms of Watts required by the device during the inference. Then, the proposed architectures would be considered to be a promising approach for real-time monocular depth estimation by offering the best trade-off between inference performances, estimation error and energy consumption, with the aim of improving the environment perception for underwater drones, lightweight robots and Internet of things.

RGB-D-Based Stair Detection and Estimation Using Deep Learning.

Stairs are common vertical traffic structures in buildings, and stair detection tasks are important in environmental perception for autonomous mobile robots. Most existing algorithms have difficulty combining the visual information from binocular sensors effectively and ensuring reliable detection at night and in the case of extremely fuzzy visual clues. To solve these problems, we propose a stair detection network with red-green-blue (RGB) and depth inputs. Specifically, we design a selective module, which can make the network learn the complementary relationship between the RGB feature maps and the depth feature maps and fuse the features effectively in different scenes. In addition, we propose several postprocessing algorithms, including a stair line clustering algorithm and a coordinate transformation algorithm, to obtain the stair geometric parameters. Experiments show that our method has better performance than existing the state-of-the-art deep learning method, and the accuracy, recall, and runtime are improved by 5.64%, 7.97%, and 3.81 ms, respectively. The improved indexes show the effectiveness of the multimodal inputs and the selective module. The estimation values of stair geometric parameters have root mean square errors within 15 mm when ascending stairs and 25 mm when descending stairs. Our method also has extremely fast detection speed, which can meet the requirements of most real-time applications.

Social Robotics in Education: A Survey on Recent Studies and Applications

Over the last years, social robots have been conceived to be used in various domains and for different purposes. This paper presents a survey on different applications and aspects of social robots in education. Different contexts are seen, ranging from children education, until higher education. A variety of applications and studies are reported, in areas like as second language tutoring, teaching assistance, and storytelling. Robotic platforms used in these applications are observed, and means of evaluation of users perception of robots are presented.

Lightweight Underwater Target Detection Algorithm Based on Dynamic Sampling Transformer and Knowledge-Distillation Optimization

Underwater robot perception is a critical task. Due to the complex underwater environment and low quality of optical images, it is difficult to obtain accurate and stable target position information using traditional methods, making it unable to meet practical use requirements. The relatively low computing power of underwater robots prevents them from supporting real-time detection with complex model algorithms for deep learning. To resolve the above problems, a lightweight underwater target detection and recognition algorithm based on knowledge distillation optimization is proposed based on the YOLOv5-lite model. Firstly, a dynamic sampling Transformer module is proposed. After the feature matrix is sparsely sampled, the query matrix is dynamically shifted to achieve the purpose of targeted attention modeling. Additionally, the shared kernel parameter convolution is used to optimize the matrix encoding and simplify the forward-propagation memory overhead. Then, a distillation method with decoupled localization and recognition is designed in the model-training process. The ability to transfer the effective localization knowledge of the positive sample boxes is enhanced, which ensures that the model maintains the same number of parameters to improve the detection accuracy. Validated by real offshore underwater image data, the experimental results show that our method provides an improvement of 6.6% and 5.0% over both baseline networks with different complexity models under the statistical index of detection accuracy mAP, which also suggests 58.8% better efficiency than models such as the standard YOLOv5. Through a comparison with other mainstream single-stage networks, the effectiveness and sophistication of the proposed algorithm are validated.

Sign Language Gesture Recognition and Classification Based on Event Camera with Spiking Neural Networks

Sign language recognition has been utilized in human–machine interactions, improving the lives of people with speech impairments or who rely on nonverbal instructions. Thanks to its higher temporal resolution, less visual redundancy information and lower energy consumption, the use of an event camera with a new dynamic vision sensor (DVS) shows promise with regard to sign language recognition with robot perception and intelligent control. Although previous work has focused on event camera-based, simple gesture datasets, such as DVS128Gesture, event camera gesture datasets inspired by sign language are critical, which poses a great impediment to the development of event camera-based sign language recognition. An effective method to extract spatio-temporal features from event data is significantly desired. Firstly, the event-based sign language gesture datasets are proposed and the data have two sources: traditional sign language videos to event stream (DVS_Sign_v2e) and DAVIS346 (DVS_Sign). In the present dataset, data are divided into five classification, verbs, quantifiers, position, things and people, adapting to actual scenarios where robots provide instruction or assistance. Sign language classification is demonstrated in spike neuron networks with a spatio-temporal back-propagation training method, leading to the best recognition accuracy of 77%. This work paves the way for the combination of event camera-based sign language gesture recognition and robotic perception for the future intelligent systems.

Morphology and Tension Perception of Cable-Driven Continuum Robots

The operation and control of the cable-driven continuum robot (CDCR) is directly determined by the actuation of cables, and the relationship between the driving length and the tension as well as the morphology of the CDCR has not been deeply researched in previous literature. In this article, a novel methodology for perceiving the morphology, tension, and driving state of the robot is proposed, which can be calculated only by the dynamic driving length/speed of cables without using shape sensors and tension sensors. First, a model of the tension distribution along with the cable path is deduced, which takes into account the influence of the sliding friction. Second, a rigid-flexible coupling dynamic model considering the strain of the cable is established. Then, a nonlinear complementary model between the driving state and the tensions of cables is constructed based on the mechanism of cable actuation, and an efficient solution algorithm is presented. Experiments of a single arm driven by two cables and a three-tandem arm driven by multiple cables were conducted to validate the proposed method. The average errors of the perceiving morphology and tension are 1.85% and 7.26%, respectively, and the transaction between the slack and tensional state of the driving cable is accurately captured. In addition, the hysteresis behavior of the cable actuation is also perceived.

The Effects of Robot Voices and Appearances on Users’ Emotion Recognition and Subjective Perception

As the influence of social robots in people’s daily lives grows, research on understanding people’s perception of robots including sociability, trust, acceptance, and preference becomes more pervasive. Research has considered visual, vocal, or tactile cues to express robots’ emotions, whereas little research has provided a holistic view in examining the interactions among different factors influencing emotion perception. We investigated multiple facets of user perception on robots during a conversational task by varying the robots’ voice types, appearances, and emotions. In our experiment, 20 participants interacted with two robots having four different voice types. While participants were reading fairy tales to the robot, the robot gave vocal feedback with seven emotions and the participants evaluated the robot’s profiles through post surveys. The results indicate that (1) the accuracy of emotion perception differed depending on presented emotions, (2) a regular human voice showed higher user preferences and naturalness, (3) but a characterized voice was more appropriate for expressing emotions with significantly higher accuracy in emotion perception, and (4) participants showed significantly higher emotion recognition accuracy with the animal robot than the humanoid robot. A follow-up study ([Formula: see text]) with voice-only conditions confirmed that the importance of embodiment. The results from this study could provide the guidelines needed to design social robots that consider emotional aspects in conversations between robots and users.

MFF-Net: Towards Efficient Monocular Depth Completion With Multi-Modal Feature Fusion

Remarkable progress has been achieved by current depth completion approaches, which produce dense depth maps from sparse depth maps and corresponding color images. However, the performances of these approaches are limited due to the insufficient feature extractions and fusions. In this work, we propose an efficient multi-modal feature fusion based depth completion framework (MFF-Net), which can efficiently extract and fuse features with different modals in both encoding and decoding processes, thus more depth details with better performance can be obtained. In specific, the encoding process contains three branches where different modals of features from both color and sparse depth input can be extracted, and a multi-feature channel shuffle is utilized to enhance these features thus features with better representation abilities can be obtained. Meanwhile, the decoding process contains two branches to sufficiently fuse the extracted multi-modal features, and a multi-level weighted combination is employed to further enhance and fuse features with different modals, thus leading to more accurate and better refined depth maps. Extensive experiments on different benchmarks demonstrate that we achieve state-of-the-art among online methods. Meanwhile, we further evaluate the predicted dense depth by RGB-D SLAM, which is a commonly used downstream robotic perception task, and higher accuracy on vehicle's trajectory can be obtained in KITTI odometry dataset, which demonstrates the high quality of our depth prediction and the potential of improving the related downstream tasks with depth completion results.

Lifelong Visual-Tactile Spectral Clustering for Robotic Object Perception

This work presents a novel visual-tactile fused clustering framework, called <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">L</u> ifelong <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">V</u> isual- <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">T</u> actile <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">S</u> pectral <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">C</u> lustering (i.e., LVTSC), to effectively learn consecutive object clustering tasks for robotic perception. Lifelong learning has become an important and hot topic in recent studies on machine learning, aiming to imitate “human learning” and reduce the computational cost when consecutively learning new tasks. Our proposed LVTSC model explores the knowledge transfer and representation correlation from a local modality-invariant perspective under modality-consistent constraint guidance. For the modality-invariant part, we design a set of modality-invariant basis libraries to capture the latent clustering centers of each modality and a set of modality-invariant feature libraries to forcibly embed the manifold information of each modality. A modal-consistent constraint reinforces the correlation between visual and tactile modalities by maximizing the feature manifold correspondences. When the object clustering task comes continuously, the overall objective is optimized by an effective alternating direction method with guaranteed convergence. Our proposed LVTSC framework has been extensively validated for its effectiveness and efficiency on the three challenging real-world robotic object perception datasets.

A mini imitation game: How individuals model social robots via behavioral outcomes and social roles

In past works, social robots have been designed to mimic human appearances and behavior. However, little is known about how human beings may imitate social robots. Drawing on social cognitive theory and the Media Equation, this study focuses on the modeling effects of social robots in an environment protection context. A lab experiment (N = 128) with a between-subjects factorial design was conducted to examine how social robots’ behavioral outcomes and social roles affected individuals’ modeling behavior. This study suggested that social robots’ positive behavioral outcomes were effective in evoking users’ modeling tendencies serially through social presence and identification or only through identification. Robots’ mere presentation of behavior with no outcomes exerted effects serially through social presence and identification. Additionally, assigning social robots an instructor role led to users’ modeling behavior serially through users’ perception of robots’ expertise and credibility. The study analyzed the psychological mechanisms behind users’ modeling behavior.

Hybrid metric-feature mapping based on camera and Lidar sensor fusion

Mapping the surroundings is a challenging task required for mobile robot. Fusion of heterogeneous sensors measurement has gradually been the main focus of robot perception. In this work, the Lidar and camera measurement is fused to obtain robust motion estimation and construct hybrid metric-feature map. By coupling visual information, the powerful tracking ability of visual features is heuristically applied to support the scan matching. With the estimation of Lidar odometry, the camera motion in complicated scene can be guaranteed. Meanwhile, the hybrid map is divided into submaps to improve the mapping efficiency, and each submap is composed of a grid-based metric submap and a group of visual features. Then a multi-stage loop closure strategy is applied to determine the loop candidate. Besides, multi-layer optimization including local bundle adjustment and pose graph optimization is performed to obtain global consistent map. The effectiveness of the proposed method is verified in different scenarios.

Robot Perception and Sensing for Environmental Awareness

Robot perception and sensing for environmental awareness is a rapidly growing field that seeks to develop robots capable of monitoring and understanding the environment in a variety of settings. This paper provides a literature review of the existing and proposed methodologies in this field, focusing on the sensing modalities and machine learning models used for environmental monitoring and management. The paper also discusses the limitations and challenges associated with these methodologies, as well as the future directions and opportunities for research. Existing methodologies in robot perception and sensing for environmental awareness include LIDAR-based mapping, RGB-D object recognition, acoustic sensing, thermal imaging, and chemical sensing. These methodologies have been applied in various contexts, such as environmental monitoring, precision agriculture, and disaster response, and have shown promise in improving our understanding of the environment and addressing environmental challenges. However, they also have limitations, such as range, resolution, sensitivity, and susceptibility to environmental factors. Proposed methodologies in robot perception and sensing for environmental awareness aim to overcome some of these limitations by integrating multiple sensing modalities and using data fusion techniques to improve accuracy and robustness. Multi-modal sensing and fusion can combine LIDAR, RGB-D cameras, microphones, and gas sensors to enable more comprehensive and accurate monitoring and assessment of environmental conditions and hazards. Future work in this field could focus on the development of more advanced and robust machine learning models, integration of more diverse and advanced sensing modalities, exploration of new applications and use cases, development of more efficient and scalable hardware and software platforms, and integration of robot sensing technologies with other data sources. Overall, robot perception and sensing for environmental awareness is an exciting and important field with significant potential to improve our understanding and management of the environment. The methodologies reviewed in this paper provide valuable insights and solutions to a wide range of environmental challenges, and future work in this field has the potential to make significant contributions to environmental science, management, and policy.

Touchformer: A Transformer-Based Two-Tower Architecture for Tactile Temporal Signal Classification.

Haptic temporal signal recognition plays an important supporting role in robot perception. This paper investigates how to improve classification performance on multiple types of haptic temporal signal datasets using a Transformer model structure. By analyzing the feature representation of haptic temporal signals, a Transformer-based two-tower structural model, called Touchformer, is proposed to extract temporal and spatial features separately and integrate them using a self-attention mechanism for classification. To address the characteristics of small sample datasets, data augmentation is employed to improve the stability of the dataset. Adaptations to the overall architecture of the model and the training and optimization procedures are made to improve the recognition performance and robustness of the model. Experimental comparisons on three publicly available datasets demonstrate that the Touchformer model significantly outperforms the benchmark model, indicating our approach's effectiveness and providing a new solution for robot perception.

Unseen Object Few-Shot Semantic Segmentation for Robotic Grasping

Semantic segmentation is a popular technology enabling robots to perceive and interact with the environment sufficiently. However, unseen objects in the new environment make it challenging to obtain accurate segmentation results. To solve this problem, we apply few-shot semantic segmentation in robot perception and propose a complete semantic grasping framework that integrates few-shot semantic segmentation and grasp pose detection for the first time. Our method can quickly identify the unseen target and obtain accurate segmentation results with only a few labeled support images. Meanwhile, our proposed multi-scale prototype extraction module and self-attention-guided feature learning module can generate representative prototypes and query features. Furthermore, we set up a few-shot semantic segmentation benchmark for robotic environments named GraspNet-10 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{i}$</tex-math></inline-formula> . Experimental results on the dataset show that our method outperforms previous state-of-the-art methods by a large margin. We also conduct real robotic grasping experiments to demonstrate the feasibility and effectiveness of our pipeline.

CARINA Project: Visual Perception Systems Applied for Autonomous Vehicles and Advanced Driver Assistance Systems (ADAS)

Autonomous mobile robots use computational techniques of great complexity so that to allow navigation in various types of dynamic environments, avoiding collisions with obstacles and always seeking to optimize the best route, ultimately enabling them to operate in a safe and precise manner. In order for navigation at this level to be possible, a variety of intelligent sensing techniques and computer vision are used. The potential of an intelligent computer vision system to detect and predict the actions of dynamic agents on the streets is applied to increase traffic safety with intelligent robotic vehicles. In this paper we present a systematic review of computer vision models for the detection and tracking of obstacles in traffic environments. Specifically, we will cover works involving 2D and 3D data fusion for both internal and external perception, as well as current trends regarding efficient model design and temporally-aware architectures. Alongside our review, we also provide a thorough discussion on the main positive and negative points of the state-of-the-art for detecting and tracking obstacles in Visual Robotic Attention works, as well as share our experience in applying visual perception for external obstacle detection and tracking, as well as internal (driver) monitoring. The results presented should serve as a compilation of the history of visual perception for autonomous mobile robots, and our contributions to the field, thus enabling an advance for research in the area of Advanced Driver Assistance Systems (ADAS) and Autonomous Vehicles.

Customers Satisfaction on Robots, Artificial Intelligence and Service Automation (RAISA) in the Hotel Industry: A Comprehensive Review

Due to the 4th Industrial Revolution, Artificial Intelligence and other internet technologies such as RAISA (Robots, Artificial Intelligence and Service Automation) have been adopted in various industries, including the hospitality and tourism industry. As the industry traditionally offers services through manual labor force, more companies adopt RAISA through a commercial service in the form of chatbots, delivery robots, robot concierge, conveyor restaurants, self-service information/check-in/check-out kiosks, and many others. The current academic resources mainly focus on customer perception or evaluation of robots themselves, and few studies link robot services with customers’ overall experience of hospitality services. This study aims to evaluate the current status of RAISA studies in the hotel industry and propose directions for future research. Thirty-seven articles were identified from Google Scholar. These articles were reviewed from four perspectives, namely, journal/year distribution, methodology, research context, theoretical foundation. Findings reveal that most studies on RAISA in the hotel industry focus on RAISA’s impacts on the hotel industry, RAISA service quality, customer acceptance of RAISA service, customer satisfaction and factors influencing Customers’ Satisfaction on RAISA Service and quantitative approach is the dominant research method. To this end, five research directions cover outstanding themes as follows: 1) strategic assessment of RAISA performance, 2) hotel financial impacts on RAISA employment, 3) customers’ satisfaction and RAISA service quality, 4) macro environment impacts on RAISA in hotels, and 5) premium determinants on customers’ acceptance on RAISA.