Perception Sensors Research Articles

Autonomous Floor and Staircase Cleaning Framework by Reconfigurable sTetro Robot with Perception Sensors

Cleaning multi-storey buildings need to be considered while developing autonomous service robots. In this paper, we introduce a novel reconfigurable platform called sTetro with the abilitiesto navigate on the floor as well as to detect then climb the staircase autonomously. To this end, an operational framework for this cleaning robot that leverages on customized deep convolution neural network (DCNN) and the RGBD camera to locate staircases in the 3D prebuilt map and then to plan trajectories by maximizing area coverage for both floor and staircase in the multi-storey environments is proposed. While building a 3D map, the staircase location is identified at the 3D point close to the center of the staircase first step using a contour detection algorithm from the boundary of the detected staircase by DCNN. The robot follows the planned trajectory to clear the floor then approaching the staircase location accurately to execute the climbing mode while cleaning the staircase to reach the next floor. The proposed methods archive the high accuracy in identifying the presence of the different staircase types, and the first step locations. Moreover, the multi-storey building evaluations have demonstrated the efficiency of the sTetro in terms of the area coverage both staircase and floor free space.

Multi-Sensor Extrinsic Calibration Using an Extended Set of Pairwise Geometric Transformations

Systems composed of multiple sensors for exteroceptive perception are becoming increasingly common, such as mobile robots or highly monitored spaces. However, to combine and fuse those sensors to create a larger and more robust representation of the perceived scene, the sensors need to be properly registered among them, that is, all relative geometric transformations must be known. This calibration procedure is challenging as, traditionally, human intervention is required in variate extents. This paper proposes a nearly automatic method where the best set of geometric transformations among any number of sensors is obtained by processing and combining the individual pairwise transformations obtained from an experimental method. Besides eliminating some experimental outliers with a standard criterion, the method exploits the possibility of obtaining better geometric transformations between all pairs of sensors by combining them within some restrictions to obtain a more precise transformation, and thus a better calibration. Although other data sources are possible, in this approach, 3D point clouds are obtained by each sensor, which correspond to the successive centers of a moving ball its field of view. The method can be applied to any sensors able to detect the ball and the 3D position of its center, namely, LIDARs, mono cameras (visual or infrared), stereo cameras, and TOF cameras. Results demonstrate that calibration is improved when compared to methods in previous works that do not address the outliers problem and, depending on the context, as explained in the results section, the multi-pairwise technique can be used in two different methodologies to reduce uncertainty in the calibration process.

Deletion of Acid-Sensing Ion Channel 3 Relieves the Late Phase of Neuropathic Pain by Preventing Neuron Degeneration and Promoting Neuron Repair.

Neuropathic pain is one type of chronic pain that occurs as a result of a lesion or disease to the somatosensory nervous system. Chronic excessive inflammatory response after nerve injury may contribute to the maintenance of persistent pain. Although the role of inflammatory mediators and cytokines in mediating allodynia and hyperalgesia has been extensively studied, the detailed mechanisms of persistent pain or whether the interactions between neurons, glia and immune cells are essential for maintenance of the chronic state have not been completely elucidated. ASIC3, a voltage-insensitive, proton-gated cation channel, is the most essential pH sensor for pain perception. ASIC3 gene expression is increased in dorsal root ganglion neurons after inflammation and nerve injury and ASIC3 is involved in macrophage maturation. ASIC currents are increased after nerve injury. However, whether prolonged hyperalgesia induced by the nerve injury requires ASIC3 and whether ASIC3 regulates neurons, immune cells or glial cells to modulate neuropathic pain remains unknown. We established a model of chronic constriction injury of the sciatic nerve (CCI) in mice. CCI mice showed long-lasting mechanical allodynia and thermal hyperalgesia. CCI also caused long-term inflammation at the sciatic nerve and primary sensory neuron degeneration as well as increased satellite glial expression and ATF3 expression. ASIC3 deficiency shortened mechanical allodynia and attenuated thermal hyperalgesia. ASIC3 gene deletion shifted ATF3 expression from large to small neurons and altered the M1/M2 macrophage ratio, thereby preventing small neuron degeneration and relieved pain.

Perception Understanding Action: Adding Understanding to the Perception Action Cycle With Spiking Segmentation.

Traditionally the Perception Action cycle is the first stage of building an autonomous robotic system and a practical way to implement a low latency reactive system within a low Size, Weight and Power (SWaP) package. However, within complex scenarios, this method can lack contextual understanding about the scene, such as object recognition-based tracking or system attention. Object detection, identification and tracking along with semantic segmentation and attention are all modern computer vision tasks in which Convolutional Neural Networks (CNN) have shown significant success, although such networks often have a large computational overhead and power requirements, which are not ideal in smaller robotics tasks. Furthermore, cloud computing and massively parallel processing like in Graphic Processing Units (GPUs) are outside the specification of many tasks due to their respective latency and SWaP constraints. In response to this, Spiking Convolutional Neural Networks (SCNNs) look to provide the feature extraction benefits of CNNs, while maintaining low latency and power overhead thanks to their asynchronous spiking event-based processing. A novel Neuromorphic Perception Understanding Action (PUA) system is presented, that aims to combine the feature extraction benefits of CNNs with low latency processing of SCNNs. The PUA utilizes a Neuromorphic Vision Sensor for Perception that facilitates asynchronous processing within a Spiking fully Convolutional Neural Network (SpikeCNN) to provide semantic segmentation and Understanding of the scene. The output is fed to a spiking control system providing Actions. With this approach, the aim is to bring features of deep learning into the lower levels of autonomous robotics, while maintaining a biologically plausible STDP rule throughout the learned encoding part of the network. The network will be shown to provide a more robust and predictable management of spiking activity with an improved thresholding response. The reported experiments show that this system can deliver robust results of over 96 and 81% for accuracy and Intersection over Union, ensuring such a system can be successfully used within object recognition, classification and tracking problem. This demonstrates that the attention of the system can be tracked accurately, while the asynchronous processing means the controller can give precise track updates with minimal latency.

Real‐time moving object detection and removal from 3D pointcloud data for humanoid navigation in dense GPS‐denied environments

AbstractRobot perception in dynamic confined unstructured environments is a challenging task due to unanticipated changes that take place in the surroundings. Although 3D perception sensors are able to capture terrain topology with high precision, the interim variations between collected sensor data that are caused due to the motion of moving entities with respect to the robot lead to noisy mappings of the environment. In this article, a real‐time 3D perception filter is presented that is capable of detecting and eliminating moving point clusters from the input pointcloud data collected in an indoor environment. Using LiDAR and IMU sensors the proposed mechanism can help in precise 3D pointcloud map generation in dynamic and unstructured GPS‐denied environments. In this article, a novel approach has been proposed based on the concepts of data clustering, relative motion, pointcloud change detection and confidence tracking. The novelty of this approach lies in its ability to detect within cluster movements and the proposal of a generic tracking method for handling inconsistent motion of objects typically found in indoor environments. For the detection of moving objects, the proposed mechanism does not require any prior knowledge about the target entity. For pointcloud preprocessing, a ground plane removal approach has been proposed based on voxel grid covariance along the axis normal to the ground. The approach was experimented on a humanoid robot in indoor office environments using Velodyne VLP‐16 LiDAR and Intel T265 IMU. The results show that the proposed approach is efficient in detecting indoor moving objects in real time.

Sequential lidar sensor system simulation: a modular approach for simulation-based safety validation of automated driving

Validating safety is an unsolved challenge before autonomous driving on public roads is possible. Since only the use of simulation-based test procedures can lead to an economically viable solution for safety validation, computationally efficient simulation models with validated fidelity are demanded. A central part of the overall simulation tool chain is the simulation of the perception components. In this work, a sequential modular approach for simulation of active perception sensor systems is presented on the example of lidar. It enables the required level of fidelity of synthetic object list data for safety validation using beforehand simulated point clouds. The elaborated framework around the sequential modules provides standardized interfaces packaging for co-simulation such as Open Simulation Interface (OSI) and Functional Mockup Interface (FMI), while providing a new level of modularity, testability, interchangeability, and distributability. The fidelity of the sequential approach is demonstrated on an everyday scenario at an intersection that is performed in reality at first and reproduced in simulation afterwards. The synthetic point cloud is generated by a sensor model with high fidelity and processed by a tracking model afterwards, which, therefore, outputs bounding boxes and trajectories that are close to reality.

Online multi-sensor calibration based on moving object tracking

Modern autonomous systems often fuse information from many different sensors to enhance their perception capabilities. For successful fusion, sensor calibration is necessary, while performing it online is crucial for long-term reliability. Contrary to currently common online approach of using ego-motion estimation, we propose an online calibration method based on detection and tracking of moving objects. Our motivation comes from the practical perspective that many perception sensors of an autonomous system are part of the pipeline for detection and tracking of moving objects. Thus, by using information already present in the system, our method provides resource inexpensive solution for the long-term reliability of the system. The method consists of a calibration-agnostic track to track association, computationally lightweight decalibration detection, and a graph-based rotation calibration. We tested the proposed method on a real-world dataset involving radar, lidar and camera sensors where it was able to detect decalibration after several seconds, while estimating rotation with error from a 20 s long scenario.

A Review and Perspective on Optical Phased Array for Automotive LiDAR

This paper aims to review the state of the art of Light Detection and Ranging (LiDAR) sensors for automotive applications, and particularly for automated vehicles, focusing on recent advances in the field of integrated LiDAR, and one of its key components: the Optical Phased Array (OPA). LiDAR is still a sensor that divides the automotive community, with several automotive companies investing in it, and some companies stating that LiDAR is a ‘useless appendix’. However, currently there is not a single sensor technology able to robustly and completely support automated navigation. Therefore, LiDAR, with its capability to map in 3 dimensions (3D) the vehicle surroundings, is a strong candidate to support Automated Vehicles (AVs). This manuscript highlights current AV sensor challenges, and it analyses the strengths and weaknesses of the perception sensor currently deployed. Then, the manuscript discusses the main LiDAR technologies emerging in automotive, and focuses on integrated LiDAR, challenges associated with light beam steering on a chip, the use of Optical Phased Arrays, finally discussing current factors hindering the affirmation of silicon photonics OPAs and their future research directions.

All-printed soft triboelectric nanogenerator for energy harvesting and tactile sensing

Rapid advancements in flexible and multifunctional electronics bring challenges to the corresponding power suppliers. Soft triboelectric nanogenerator (TENG), as a promising route to satisfy the demands of various self-powered sources and energy harvesting devices, has been developed in this study via facile all-printing method. The nanogenerator features a thin film with micro patterned friction silicone layers, which is fabricated through direct ink writing technology. Wrinkled silver electrode is sandwiched and protected by silicone layers, and constructed through prestrain, release, and buckling process based on inkjet printing technology. As an experimental result, TENG with 100% prestrain produces the maximum output voltage of 44.16 V at the frequency of 5 Hz and external force of 10 N, higher than the pristine nanogenerator. The output voltage of these TENGs could be enhanced slightly when it is stretched to a certain degree. The as-prepared TENG can work for biomechanical energy harvesting driven by human motions and instantaneously power 20 commercial Light Emitting Diodes (LEDs). Moreover, the all-printed soft TENG arrays show potential applications in tactile sensor for touch perception. Our work opens a new idea for fabrication of flexible and wearable TENG based on printing technology to shorten fabrication cycle, simplify individual design and make it more accessible for practical application. In this work, we present an all-printed soft triboelectric nanogenerator that is designed to harvest small mechanical energy. Based on the direct ink writing technology, the friction layer can be fabricated with regular micro structures to enlarge the surface area, and to protect the sandwiched electrode inside as well. The wrinkled silver electrode is constructed on the prestretched friction layer according to the inkjet printing process. The assembled TENG with an activation area of 10 × 10 mm 2 , operating in the contact-separation mode, can achieve an output voltage of 44.16 V at the prestrain 100% with the applied force of 10 N and frequency of 5 Hz. A power density of 1.03 W/m 2 could be produced correspondingly to drive 10 commercial LEDs connected in series. In addition, TENG can also be applied to monitor the movements of waist, foot and elbow for biomechanical energy harvesting, and the printed arrays have the potential application in mapping the touch location as tactile sensor. This simple, low-cost, controllable and reliable approach to TENG device based on all-printing technology provides new dimensions for rapid manufacturing of energy harvesters and wearable self-powered sensors. • Flexible and stretchable triboelectric nanogenerator was proposed based on all-printing method. • Micropatterned friction surface was constructed via direct ink writing. • Wrinkled silver electrode was fabricated through inkjet printing. • The powerful TENG is a promising candidate for portable and wearable biosensors.

Mixed-reality Automotive Testing with SENSORIS

Highly automated and autonomous vehicles become more and more widespread changing the classical way of testing and validation. Traditionally, the automotive industry has pursued testing rather in real-world or in pure virtual simulation environments. As a new possibility, mixed-reality testing has also appeared enabling an efficient combination of real and simulated elements of testing. Furthermore, vehicles from different OEMs will have a common interface to communicate with a test system. The paper presents a mixed-reality test framework for visualizing perception sensor feeds real-time in the Unity 3D game engine. Thereby, the digital twin of the tested vehicle and its environment are realized in the simulation. The communication between the sensors of the tested vehicle and the central computer running the test is realized via the standard SENSORIS interface. The paper outlines the hardware and software requirements towards such a system in detail. To show the viability of the system a vehicle in the loop test has been carried out.

A Real-Time Infrared Stereo Matching Algorithm for RGB-D Cameras’ Indoor 3D Perception

Low-cost, commercial RGB-D cameras have become one of the main sensors for indoor scene 3D perception and robot navigation and localization. In these studies, the Intel RealSense R200 sensor (R200) is popular among many researchers, but its integrated commercial stereo matching algorithm has a small detection range, short measurement distance and low depth map resolution, which severely restrict its usage scenarios and service life. For these problems, on the basis of the existing research, a novel infrared stereo matching algorithm that combines the idea of the semi-global method and sliding window is proposed in this paper. First, the R200 is calibrated. Then, through Gaussian filtering, the mutual information and correlation between the left and right stereo infrared images are enhanced. According to mutual information, the dynamic threshold selection in matching is realized, so the adaptability to different scenes is improved. Meanwhile, the robustness of the algorithm is improved by the Sobel operators in the cost calculation of the energy function. In addition, the accuracy and quality of disparity values are improved through a uniqueness test and sub-pixel interpolation. Finally, the BundleFusion algorithm is used to reconstruct indoor 3D surface models in different scenarios, which proved the effectiveness and superiority of the stereo matching algorithm proposed in this paper.

Fault Detection, Isolation, Identification and Recovery (FDIIR) Methods for Automotive Perception Sensors Including a Detailed Literature Survey for Lidar.

Perception sensors such as camera, radar, and lidar have gained considerable popularity in the automotive industry in recent years. In order to reach the next step towards automated driving it is necessary to implement fault diagnosis systems together with suitable mitigation solutions in automotive perception sensors. This is a crucial prerequisite, since the quality of an automated driving function strongly depends on the reliability of the perception data, especially under adverse conditions. This publication presents a systematic review on faults and suitable detection and recovery methods for automotive perception sensors and suggests a corresponding classification schema. A systematic literature analysis has been performed with focus on lidar in order to review the state-of-the-art and identify promising research opportunities. Faults related to adverse weather conditions have been studied the most, but often without providing suitable recovery methods. Issues related to sensor attachment and mechanical damage of the sensor cover were studied very little and provide opportunities for future research. Algorithms, which use the data stream of a single sensor, proofed to be a viable solution for both fault detection and recovery.

Automotive Lidar Modelling Approach Based on Material Properties and Lidar Capabilities.

Development and validation of reliable environment perception systems for automated driving functions requires the extension of conventional physical test drives with simulations in virtual test environments. In such a virtual test environment, a perception sensor is replaced by a sensor model. A major challenge for state-of-the-art sensor models is to represent the large variety of material properties of the surrounding objects in a realistic manner. Since lidar sensors are considered to play an essential role for upcoming automated vehicles, this paper presents a new lidar modelling approach that takes material properties and corresponding lidar capabilities into account. The considered material property is the incidence angle dependent reflectance of the illuminated material in the infrared spectrum and the considered lidar property its capability to detect a material with a certain reflectance up to a certain range. A new material classification for lidar modelling in the automotive context is suggested, distinguishing between 7 material classes and 23 subclasses. To measure angle dependent reflectance in the infrared spectrum, a new measurement device based on a time of flight camera is introduced and calibrated using Lambertian targets with defined reflectance values at , , and . Reflectance measurements of 9 material subclasses are presented and 488 spectra from the NASA ECOSTRESS library are considered to evaluate the new measurement device. The parametrisation of the lidar capabilities is illustrated by presenting a lidar measurement campaign with a new Infineon lidar prototype and relevant data from 12 common lidar types.

Fog Classification by Their Droplet Size Distributions: Application to the Characterization of Cerema’s Platform

Fog is one of major challenges for transportation systems. The automation of the latter is based on perception sensors that can be disrupted by atmospheric conditions. As fog conditions are random and non-reproducible in nature, Cerema has designed a platform to generate fog and rain on demand. Two types of artificial fog with different droplet size distributions are generated: they correspond to radiation fogs with small and medium droplets. This study presents an original method for classifying these different types of fog in a descriptive and quantitative way. It uses a new fog classification coefficient based on a principal component analysis, which measures the ability of a pair of droplet size distribution descriptors to differentiate between the two different types of fog. This method is applied to a database containing more than 12,000 droplet size distributions collected within the platform. It makes it possible to show: (1) that the two types of fog proposed by Cerema have significantly different droplet size distributions, for meteorological visibility values from 10 m to 1000 m; (2) that the proposed droplet size distribution range is included in the natural droplet size distribution range; (3) that the proposed droplet size distribution range should be extended in particular with larger droplets. Finally, the proposed method makes it possible to compare the different fog droplet size distribution descriptors proposed in the literature.

Path-Following With LiDAR-Based Obstacle Avoidance of an Unmanned Surface Vehicle in Harbor Conditions

This article studies the design, modeling, and implementation challenges of a path-following with obstacle avoidance algorithms as guidance, navigation, and control (GNC) architecture of an unmanned surface vehicle (USV) in harbor conditions. First, an effective mathematical model is developed based on system identification, validating the USV model with field-test data. Then, a guidance system is addressed based on a line-of-sight algorithm, which uses a LiDAR as the main perception sensor for the obstacle avoidance algorithm. The GNC architecture uses a modular approach, including obstacle detection, path-following, and control in the USV platform. Finally, an implementation challenge in two control scenarios, simulation and field test, is addressed to validate the designed GNC architecture.

A Framework to Analyze Noise Factors of Automotive Perception Sensors

Automated vehicles (AVs) are one of the breakthroughs of this century. The main argument to support their development is increased safety and reduction of human and economic losses; however, to demonstrate that AVs are safer than human drivers billions of miles of testing are required. Thus, realistic simulation and virtual testing of AV systems and sensors are crucial to accelerate the technological readiness. In particular, perception sensor measurements are affected by uncertainties due to noise factors; these uncertainties need to be included in simulations. This letter presents a framework to exhaustively analyze and simulate the effect of the combination of noise factors on sensor data. We applied the framework to analyze one sensor, the light detection and ranging (LiDAR), but it can be easily adapted to study other sensors. Results demonstrate that single noise factor analysis gives an incomplete knowledge of measurement degradation and perception is dramatically hindered when more noises are combined. The proposed framework is a powerful tool to predict the degradation of AV sensor performance.

Three-Dimensional Structured Dual-Mode Flexible Sensors for Highly Sensitive Tactile Perception and Noncontact Sensing.

This work reports a three-dimensional (3D) structured multifunctional sensor by connecting a magnetowhisker with a superflexible patterned skin film. Composed of percolation networks of silver nanowires, the patterned skin film is integrated via a simple template manufacturing method without increasing the complexity and sacrificing the flexibility. The as-prepared 3D structured sensor can realize the multimodal detection of out-of-plane tactile stimuli and details of noncontact environmental obstacles in multiple directions. Here, the sensor's perception behaviors on compression, pulling, magnetic field, sound waves, airflow, water level, water flow, and backwash are presented. Furthermore, the 3D structured sensor obtains outstanding mechanical robustness and stability for 8000 cycles, excellent sensitivity (12 800% when the applied pulling displacement was 3.5 mm; 152% T-1 when the magnetic flux density variation was 40.6 mT), ultrahigh response time, and ultrahigh recovery time (∼5 ms), which may meet the industrial sensing requirement for artificial tactile electronics. Facile manufacturing processes and outstanding multimodal sensing characteristics make the 3D structured sensor to possess great potential to be implemented in the next-generation intelligent bionic equipment or systems.

Autonomous collision detection and avoidance for ARAGON USV: Development and field tests

AbstractThis study addresses the development of algorithms for multiple target detection and tracking in the framework of sensor fusion and its application to autonomous navigation and collision avoidance systems for the unmanned surface vehicle (USV) Aragon. To provide autonomous navigation capabilities, various perception sensors such as radar, lidar, and cameras have been mounted on the USV platform and automatic ship detection algorithms are applied to the sensor measurements. The relative position information between the USV and nearby objects is obtained to estimate the motion of the target objects in a sensor‐level tracking filter. The estimated motion information from the individual tracking filters is then combined in a central‐level fusion tracker to achieve persistent and reliable target tracking performance. For automatic ship collision avoidance, the combined track data are used as obstacle information, and appropriate collision avoidance maneuvers are designed and executed in accordance with the international regulations for preventing collisions at sea (COLREGs). In this paper, the development processes of the vehicle platform and the autonomous navigation algorithms are described, and the results of field experiments are presented and discussed.

Improving Visual Localization Accuracy in Dynamic Environments Based on Dynamic Region Removal

Visual localization is a fundamental capability in robotics and has been well studied for recent decades. Although many state-of-the-art algorithms have been proposed, great success usually builds on the assumption that the working environment is static. In most of the real scenes, the assumption cannot hold because there are inevitably moving objects, especially humans, which significantly degrade the localization accuracy. To address this problem, we propose a robust visual localization system building on top of a feature-based visual simultaneous localization and mapping algorithm. We design a dynamic region detection method and use it to preprocess the input frame. The detection process is achieved in a Bayesian framework which considers both the prior knowledge generated from an object detection process and observation information. After getting the detection result, feature points extracted from only the static regions will be used for further visual localization. We performed the experiments on the public TUM data set and our recorded data set, which shows the daily dynamic scenarios. Both qualitative and quantitative results are provided to show the feasibility and effectiveness of the proposed method. Note to Practitioners —This article was motivated by the visual localization problem of mobile robots in dynamic working environments. Visual localization is an important and fundamental capability in robotics. When a mobile robot is in an unknown environment, one important thing is to localize itself using the data collected from the perception sensors. In real scenes, the current localization algorithms often suffer from moving objects. The existence of moving objects, especially humans, brings a lot of noise into the localization process, which poses a big challenge and makes the robotic implementation unstable. In this article, a novel strategy is designed to handle this problem. We leverage both the object detection result and observation information in a Bayesian framework for dynamic region detection. Once the dynamic regions are determined, we discard them and feed the other regions in a state-of-the-art visual simultaneous localization and mapping system for further visual localization. The proposed strategy greatly improves the localization accuracy in dynamic working environments and guarantees the robustness for robotic implementation.

MPC-Based Energy Management Strategy for an Autonomous Hybrid Electric Vehicle

Despite the current intense research on each of the subjects of electrification and autonomous driving, potential advantages as a result of the interaction of these two mainstreams in automotive have not been effectively studied yet. Autonomous vehicles generate an unprecedented amount of real-time data due to excessive use of perception sensors and processing units. In this article, we present a novel approach for improving the fuel economy of an autonomous hybrid electric vehicle by taking advantage of this qrydata. We introduce the term of autonomous-specific energy management strategy (ASEMS) and we present an example of such a strategy using model predictive control (MPC). Specifically, we show how a more fuel-optimal energy management strategy (EMS) can be achieved for the power-split powertrain of an autonomous hybrid electric vehicle using the motion planning data. We use an optimization-based motion planning approach and feed the resulting velocity profile up to the prediction horizon to the MPC-based EMS. The presented approach shows 2% to 12.81% less fuel consumption for the two extreme cases of 100 and 1000 meters as the prediction horizons, compared to a rule-based EMS. The presented EMS fuel-optimality for the 1000 meters is only 6.91% sub-optimal compared to the globally optimal results of dynamic programming.