Perception Sensors Research Articles

Dynamic Fuzzy Background Removal for Indoor Human Target Perception Based on Thermopile Array Sensor

With the ageing of population, timely assistance for the living-alone elderly is becoming a worldwide problem. The precondition of assistance is to correctly and timely perceive human behavior. The thermopile array sensor is one of the most promising perception sensors, similar to a low-resolution infrared imaging camera. Besides humans, computers and many other objects indoors emit infrared radiation and hence can be captured by the infrared thermopile array sensor at the same time. The background of the infrared image of sensor is complex and dynamically changing, and we should remove the dynamic background, i.e. to remove non-human thermal disturbances first. Based on a thermopile array sensor MLX90640, with an array output of $24\times32$ pixels, this paper proposes a dynamic fuzzy spatiotemporal background removal (DFBR) for human activity perception. Each pixel of the sensor’s output has its own dynamic parameters, background temperature ( ${{BT}}$ ) and temperature threshold ( ${{TT}}$ ), which are used to dynamically determine this pixel to be removed or not. A fuzzy background membership function is proposed to update each pixel’s ${{TT}}$ and ${{BT}}$ based on background change and temperature value below the human body. The pixel whose temperature is lower than the sum of its ${{TT}}$ and ${{BT}}$ would be removed. The corrosion is used to make the infrared image clear after background removal. Moreover, a foreground complement is proposed to complement the misremoval due to the thermal disturbance. The experimental results have shown that the DFBR can remove the background of low-resolution infrared image online dynamically and correctly.

INSPEX: Optimize Range Sensors for Environment Perception as a Portable System.

Environment perception is crucial for the safe navigation of vehicles and robots to detect obstacles in their surroundings. It is also of paramount interest for navigation of human beings in reduced visibility conditions. Obstacle avoidance systems typically combine multiple sensing technologies (i.e., LiDAR, radar, ultrasound and visual) to detect various types of obstacles under different lighting and weather conditions, with the drawbacks of a given technology being offset by others. These systems require powerful computational capability to fuse the mass of data, which limits their use to high-end vehicles and robots. INSPEX delivers a low-power, small-size and lightweight environment perception system that is compatible with portable and/or wearable applications. This requires miniaturizing and optimizing existing range sensors of different technologies to meet the user’s requirements in terms of obstacle detection capabilities. These sensors consist of a LiDAR, a time-of-flight sensor, an ultrasound and an ultra-wideband radar with measurement ranges respectively of 10 m, 4 m, 2 m and 10 m. Integration of a data fusion technique is also required to build a model of the user’s surroundings and provide feedback about the localization of harmful obstacles. As primary demonstrator, the INSPEX device will be fixed on a white cane.

Multi-sensor Fusion Road Friction Coefficient Estimation During Steering with Lyapunov Method.

The road friction coefficient is a key parameter for autonomous vehicles and vehicle dynamic control. With the development of autonomous vehicles, increasingly, more environmental perception sensors are being installed on vehicles, which means that more information can be used to estimate the road friction coefficient. In this paper, a nonlinear observer aided by vehicle lateral displacement information for estimating the road friction coefficient is proposed. First, the tire brush model is modified to describe the tire characteristics more precisely in high friction conditions using tire test data. Then, on the basis of vehicle dynamics and a kinematic model, a nonlinear observer is designed, and the self-aligning torque of the wheel, lateral acceleration, and vehicle lateral displacement are used to estimate the road friction coefficient during steering. Finally, slalom tests and DLC (Double Line Change) tests in high friction conditions are conducted to verify the proposed estimation algorithm. Test results showed that the proposed method performs well during steering and the estimated road friction coefficient converges to the reference value rapidly.

Quantifying what goes unseen in instrumented and autonomous vehicle perception sensor data – A case study

It is becoming common to use instrumented probe vehicles (IPV) equipped with perception sensors (e.g., LIDAR or radar) to monitor ambient traffic for empirical studies of traffic dynamics and driver behavior, while the rapidly advancing field of autonomous vehicles (AV) uses perception sensors to identify and mitigate hazards. Yet the perception sensors are far from perfect, so targets of interest go undetected. The “unseen” targets might lead to inaccurate theories of traffic flow or worse, autonomous vehicle crashes. The challenge with the unseen targets is that it is difficult to detect the absence. There is little study of how vehicle-mounted perception sensors perform “in the wild” because often the perception sensors themselves usually provide the best measurements available. The focus of this work is IPV data for offline study, but the general findings are also relevant for perception sensors used in AV applications.While there is a growing number of IPV data sets, to date most suffer from the lack of independent validation of the perception sensors. This paper fuses individual vehicle actuations from loop detectors and concurrent ambient vehicle trajectories collected from an IPV to provide a rare opportunity to assess the perception sensor performance in situ, using the loop detector data to “see” what is missed by the perception sensors. While the specific results are unique to the IPV, one should expect similar biases from other perception sensor data; though without an independent measure like the loop detectors in this study, it will be difficult to quantify what goes unseen in those data sets. These findings also highlight the need to develop means to actively calibrate mobile perception sensors for AV's. It is important to capture the behavior of these systems in situ, as maintained by private owners, though we envision in the case of an AV the role of the loop detectors might be replaced with localization information from other AV's. The data fusion has added benefits for the study of traffic dynamics, with the loop detectors providing continuous monitoring at the detector locations and the IPV providing spatially rich information about the ambient traffic as it travels between detector stations. This fused data set will be available upon publication.

Augmented Vehicular Reality

Autonomous vehicle prototypes today come with line-of-sight depth perception sensors like 3D cameras. These 3D sensors are used for improving vehicular safety in autonomous driving, but have fundamentally limited visibility due to occlusions, sensing range, and extreme weather and lighting conditions. To improve visibility and performance, we explore a capability called Augmented Vehicular Reality (AVR). AVR broadens the vehicle's visual horizon by enabling it to wirelessly share visual information with other nearby vehicles. We show that AVR is feasible using off-the-shelf wireless technologies, and it can qualitatively change the decisions made by autonomous vehicle path planning algorithms. Our AVR prototype achieves positioning accuracies that are within a few percentages of car lengths and lane widths, and it is optimized to process frames at 30fps.

A welding seam identification method based on cross-modal perception

PurposeAs an automatic welding process may experience some disturbances caused by, for example, splashes and/or welding fumes, misalignments/poor positioning, thermally induced deformations, strong arc lights and diversified welding joints/grooves, precisely identifying the welding seam has a great influence on the welding quality. This paper aims to propose a robust method for identifying this seam based on cross-modal perception.Design/methodology/approachFirst, after a welding image obtained from a structured-light vision sensor (here laser and vision are integrated into a cross-modal perception sensor) is filtered, in a sufficiently small area, the extended Kalman filter is used to prevent possible disturbances to search for its laser stripe. Second, to realize the extraction of the profile of welding seam, the least square method is used to fit a sequence of centroids determined by the scanning result of columns displayed on the tracking window. Third, this profile is then qualitatively described and matched using a proposed character string method.FindingsIt is demonstrated that it maintains real time and is clearly superior in terms of accuracy and robustness, though its real-time performance is not the best.Originality/valueThis paper proposes a robust method for automatically identifying and tracking a welding seam.

Body Heat Powers Future Electronic Skins

Ruoming Tian recently joined the Mark Wainwright Analytical Centre as an XRD research assistant at the University of New South Wales. She was formerly a Research Scientist at the Toyota Physical and Chemical Research Institute in Japan (2015–2018) and received her PhD from the University of New South Wales in 2014. Her research interests focus on the nano- and micro-materials, flexible hybrid materials, and smart devices for harvesting both high- and low-grade thermal energy. She has published 2 book chapters and 18 peer-reviewed articles and delivered more than 15 talks at international conferences and leading research institutes. Yuqing Liu is currently an Associate Research Fellow and received her PhD (2018) in the Intelligent Polymer Research Institute (IPRI), University of Wollongong, Australia. Her research interest is on the fabrication of wearable energy storage and conversion devices (e.g. thermos-cells, batteries, supercapacitors, etc.), mainly focusing on nanostructured electrode materials, gel electrolyte/electrode interface, and device fabrication via printing techniques. Kunihito Koumoto, Professor Emeritus of Nagoya University, received his BS, MS, and PhD from the University of Tokyo (UT). He served as an Assistant Professor, Lecturer, and Associate Professor at UT (1979–1992) and a full professor at Nagoya University (1992–2015). He has published 495 papers and reviews and 58 book chapters in inorganic materials chemistry. He was awarded Medal with Purple Ribbon by the Emperor of Japan in 2013 and Outstanding Achievement Award by the International Thermoelectric Society in 2018. His current research focuses on inorganic/organic nanohybrid thermoelectrics for energy harvesting. Koumoto can be reached by email at koumoto@apchem.nagoya-u.ac.jp . Jun Chen is currently a Full Professor at IPRI|ACES, Australian Institute for Innovative Materials, University of Wollongong (UOW). He received BE from Zhejiang University of Technology (1995) and PhD from UOW (2003). Professor Chen has published more than 180 papers and has been named a Highly Cited Researcher for 2018 (Clarivate Analytics). His research interests include sustainable energy devices, electro-/bio-interfaces, nano-/micro-materials, 2D/3D printing, and smart wearable electronic devices. Apart from academic achievements, Chen has also been involved in two spin-off companies: AQUAHYDREX as one of the key inventors, and IMAGINE as one of the scientific founders.

Multimodal Sensor Fusion for Robust Obstacle Detection and Classification in the Maritime RobotX Challenge

This paper describes a novel probabilistic sensor fusion framework aimed at improving obstacle detection accuracy and classification of various targets experienced in the Maritime RobotX Challenge. In both the 2014 and 2016 Maritime RobotX Challenges, it was found that detecting obstacles using LIDAR only and classifying obstacles using vision only can be challenged by environmental conditions, such as glare from the sun or by objects such as the spherical black buoys from the obstacle field that disperses LIDAR rays. In this paper, a new multimodal sensor fusion approach is proposed that combines data streams from perception sensors, such as LIDAR, RADAR, and cameras, to improve the robustness of detection and classification performance over a single sensor method. Using data collected from both the 2014 and 2016 Maritime RobotX Challenges, an evaluation of the perception framework is provided. The proposed detection and classification framework is now being transferred to the queensland university of technology (QUT) autonomous surface vehicle to improve overall mapping accuracy and task execution.

A Probability Occupancy Grid Based Approach for Real-Time LiDAR Ground Segmentation

Autonomous vehicles have been emerging over the past few years because of the sophisticated processing of data from different types of perception sensors, such as LiDAR, radar, and camera. Ground segmentation plays an important role in the sequence of data processing for environment perception tasks, as it can help to reduce the size of data to be processed and further decrease the overall computational time. However, the over-segmentation, under-segmentation, or slow-segmentation on non-flat surface usually occurs due to the characteristics of the 3D LIDAR data, such as occlusion in complex urban environment. To address these problems, in this paper, we proposed a probability occupancy grid-based approach for real-time ground segmentation by using a single LiDAR sensor. The effectiveness and robustness of our proposed method are evaluated and demonstrated by the real-time experiments that span different traffic scenarios from heavy traffic to light traffic.

Visualizing Cellular Gibberellin Levels Using the nlsGPS1 Förster Resonance Energy Transfer (FRET) Biosensor.

The phytohormone gibberellin (GA) is a small, mobile signaling molecule that plays a key role in seed germination, cellular elongation, and developmental transitions in plants. Gibberellin Perception Sensor 1 (GPS1) is the first Förster resonance energy transfer (FRET)-based biosensor that allows monitoring of cellular GA levels in vivo. By measuring a fluorescence emission ratio of nuclear localized-GPS1 (nlsGPS1), spatiotemporal mapping of endogenously and exogenously supplied GA gradients in different tissue types is feasible at a cellular scale. This protocol will describe how to image nlsGPS1 emission ratios in three example experiments: steady-state, before-and-after exogenous gibberellin A4 (GA4) treatments, and over a treatment time-course. We also provide methods to analyze nlsGPS1 emission ratios using both Fiji and a commercial three-dimensional (3-D) micrograph visualization and analysis software and explain the limitations and likely pitfalls of using nlsGPS1 to quantify gibberellin levels.

Human-Machine Interface for a Smart Wheelchair

The paper describes the integration of hardware and software with sensor technology and computer processing to develop the next generation intelligent wheelchair. The focus is a computer cluster design to test high performance computing for smart wheelchair operation and human interaction. The LabVIEW cluster is developed for real-time autonomous path planning and sensor data processing. Four small form factor computers are connected over a Gigabit Ethernet local area network to form the computer cluster. Autonomous programs are distributed across the cluster for increased task parallelism to improve processing time performance. The distributed programs operating frequency for path planning and motion control is 50Hz and 12.3Hz for 0.3 megapixel robot vision system. To monitor the operation and control of the distributed LabVIEW code, network automation is integrated into the cluster software along with a performance monitor. A link between the computer motion control program and the wheelchair joystick control of the drive train is developed for the computer control interface. A perception sensor array and control circuitry is integrated with the computer system to detect and respond to the wheelchair environment. Multiple cameras are used for image processing and scanning laser rangefinder sensors for obstacle avoidance in the cluster program. A centralized power system is integrated to power the smart wheelchair along with the cluster and sensor feedback system. The on board computer system is evaluated for cluster processing performance for the smart wheelchair, incorporating camera machine vision and LiDAR perception for terrain obstacle detection, operating in urban scenarios.

Learning environment for robotics education and industry-academia collaboration

Abstract It is expected that by utilizing digital technologies, advanced robotics and artificial intelligence, the manufacturing base of Europe will become stronger and allow production re-shoring from other trade areas to take place. The European competitiveness is tied to better competences of the workforce and fast implementation of new technologies. This requires new approaches for formal and non-formal education. For this, we propose a new robotics learning concept and collaboration scheme to support both MSc level education, but also non-formal education with industry. The non-formal education example could be a combination of an education package followed by rapid experimenting with a robot system. In order to facilitate the learning process, we have established the Tampere RoboLab and joint academia-industry education modules for both formal and non-formal education. The Tampere RoboLab operates with similar principles as e.g. Fab Labs (fabrication laboratories), but the focus is on indoor stationary and mobile robotics. Aside from education, the concept allows system interoperability testing and pre-competitive research to be done in the same premises as well as field robotics by providing the state of art localisation and perception sensors, and computation and communication devices. This paper will introduce the concept, used hardware and software configurations, education modules and the forms of industry-academia collaboration.

Fast real-time localization with sparse digital maps for connected automated vehicles in urban areas

Abstract For the realization of advanced automation systems in road vehicles, accurate and robust localization is a crucial requirement. Satellite-based systems such as GPS are generally able to provide geolocations, but their precision and robustness can be impaired strongly due to shading of satellites and multipath effects, especially in urban surroundings. Localization methods based on environment perception sensors and digital maps are therefore widely used in the field of autonomous vehicles with accuracy, robustness, real-time capability and sparseness of the maps being major objectives. In this paper, we present a fast, real-time capable implementation of a Monte Carlo Localization scheme which operates on a storage space efficient digital map and is targeted to provide precise localization in urban surroundings at a rate of 50 Hz. For the experimental evaluation, we use our test vehicle’s LiDAR sensor combined with wheel odometry, inertial measurements and a low-cost GPS.

Evidence Filter of Semantic Segmented Image From Around View Monitor in Automated Parking System

An Around View Monitor (AVM) is widely used as one of the perception sensors for automated parking systems. By applying semantic segmentation based on a deep learning approach, the AVM can detect two essential elements for automated parking systems: slot marking and obstacles. However, the perception based on the deep learning approach in the AVM has certain limitations such as occlusion of the ego-vehicle region, distortion of 3D objects, and environmental noise. We overcome the problems by proposing an evidence filter that improves the detection performance based on evidence theory and a Simultaneous Localization and Mapping (SLAM) algorithm. The proposed algorithm is composed of three parts: the semantic segmentation of the AVM image, confidence modeling based on evidence theory, and evidence SLAM. Semantic segmentation classifies the grids in the AVM image into three states: slot marking, freespace, and obstacle. The grids with these three states are modeled by a confidence model based on evidence theory. Finally, the states of the grids around the ego-vehicle are accumulated and estimated by the evidence SLAM. The proposed filter was evaluated by experiments in real parking-lot environments.

Lane Endpoint Detection and Position Accuracy Evaluation for Sensor Fusion-Based Vehicle Localization on Highways.

Landmark-based vehicle localization is a key component of both autonomous driving and advanced driver assistance systems (ADAS). Previously used landmarks in highways such as lane markings lack information on longitudinal positions. To address this problem, lane endpoints can be used as landmarks. This paper proposes two essential components when using lane endpoints as landmarks: lane endpoint detection and its accuracy evaluation. First, it proposes a method to efficiently detect lane endpoints using a monocular forward-looking camera, which is the most widely installed perception sensor. Lane endpoints are detected with a small amount of computation based on the following steps: lane detection, lane endpoint candidate generation, and lane endpoint candidate verification. Second, it proposes a method to reliably measure the position accuracy of the lane endpoints detected from images taken while the camera is moving at high speed. A camera is installed with a mobile mapping system (MMS) in a vehicle, and the position accuracy of the lane endpoints detected by the camera is measured by comparing their positions with ground truths obtained by the MMS. In the experiment, the proposed methods were evaluated and compared with previous methods based on a dataset acquired while driving on 80 km of highway in both daytime and nighttime.

FAST Pre-Filtering-Based Real Time Road Sign Detection for Low-Cost Vehicle Localization.

In order to overcome the limitations of GNSS/INS and to keep the cost affordable for mass-produced vehicles, a precise localization system fusing the estimated vehicle positions from low-cost GNSS/INS and low-cost perception sensors is being developed. For vehicle position estimation, a perception sensor detects a road facility and uses it as a landmark. For this localization system, this paper proposes a method to detect a road sign as a landmark using a monocular camera whose cost is relatively low compared to other perception sensors. Since the inside pattern and aspect ratio of a road sign are various, the proposed method is based on the part-based approach that detects corners and combines them to detect a road sign. While the recall, precision, and processing time of the state of the art detector based on a convolutional neural network are 99.63%, 98.16%, and 4802 ms respectively, the recall, precision, and processing time of the proposed method are 97.48%, 98.78%, and 66.7 ms, respectively. The detection performance of the proposed method is as good as that of the state of the art detector and its processing time is drastically reduced to be applicable for an embedded system.

A Perception-Based Fuzzy Route Planing Algorithm for Autonomous Unmanned Ground Vehicles

Path planning plays an integral role in mission planning for ground vehicle operations in urban areas. Determining the optimum path through an urban area is a well-understood problem for traditional ground vehicles; however, in the case of autonomous unmanned ground vehicles (UGVs), additional factors must be considered. For an autonomous UGV, perception algorithms rather than platform mobility will be the limiting factor in operational capabilities. For this study, perception was incorporated into the path planning process by associating sensor error costs with traveling through nodes within an urban road network. Three common perception sensors were used for this study: GPS, LIDAR, and IMU. Multiple set aggregation operators were used to blend the sensor error costs into a single cost, and the effects of choice of aggregation operator on the chosen path were observed. To provide a robust path planning ability, a fuzzy route planning algorithm was developed using membership functions and fuzzy rules to allow for qualitative route planning in the case of generalized UGV performance. The fuzzy membership functions were then applied to several paths through the urban area to determine what sensors were optimized in each path to provide a measure of the UGV’s performance capabilities. The research presented in this paper shows the impacts that sensing/perception has on ground vehicle route planning by demonstrating a fuzzy route planning algorithm constructed by using a robust rule set that quantifies these impacts.

Design of Sensor Data Fusion Algorithm for Mobile Robot Navigation Using ANFIS and Its Analysis Across the Membership Functions

Design and development of autonomous mobile robots attracts more attention in the era of autonomous navigation. There are various algorithms used in practice for solving research problems related to the robot model and its operating environment. This paper presents the design of data fusion algorithm using Adaptive Neuro Fuzzy Interface (ANFIS) for the navigation of mobile robots. Detailed analysis of various membership functions (MFs) provided in this paper helps to select the most appropriate MF for the design of similar navigation systems. The combined use of fuzzy and neural networks in ANFIS makes the measured distance value of the residual covariance consistent with its actual value. The data fusion algorithm within the controller of the mobile robot fuses the input from ultrasonic and infrared sensors for better environment perception. The results indicate that the data fusion algorithm provides minimal root mean square error (RMSE) and mean absolute percentage error (MAPE) when compared with that of the individual sensors.

Dynamic 3-D Reconstruction of Human Targets via an Omni-Directional Thermal Sensor

We explore the dynamic 3-D reconstruction (D3DR) of the target view in real-time images from the omni-directional (O-D) thermal sensor for intelligent perception of robotic systems. Recent O-D 3-D reconstruction methodologies are mainly focused on O-D visible-band vision for localization, mapping, calibration, and tracking, but there is no significant research for thermal O-D. The 3-D reconstruction from O-D images and the use of O-D thermal vision have not been sufficiently addressed. The thermal O-D images do not provide sharp-edge boundaries as in color vision cameras due to texture and mirror distortion. In order to fully address O-D thermal 3-D reconstruction, we proposed the D3DR method that dynamically detects the target region and densely reconstructs the detected target region to solve the non-sharp-edge boundaries’ issue. We analyzed several different imaging positions, different baseline distances, and target distances with respect to the robot position for the best coverage of the target view with a minimum reconstruction error. We also look at the optimum number of observations for reconstruction using an optimization to find the compromise between accuracy, methodology, and number of observations. The benefits of this method are the accurate distance of the target from the camera, high accuracy, and low computation time of 3-D reconstruction.

Track Fusion and Behavioral Reasoning for Moving Vehicles Based on Curvilinear Coordinates of Roadway Geometries

This paper presents track fusion and behavioral reasoning for moving vehicles in close proximity based on the curvilinear coordinates of roadway geometries. The inferred track and behavior of other vehicles can be used to perform safe actions in intelligent vehicle applications and autonomous driving. Vehicle detections from multiple perception sensors are integrated using track-to-track (T2T) fusion based on a cross-covariance method, and this T2T fusion is performed with curvilinear coordinates created using prebuilt roadway geometry on a digital map. The coordinate conversion to curvilinear space has many benefits for behavioral reasoning and tracking, such as constraining problem spaces and dimensions. A machine learning classifier based on a support vector machine is then applied to deduce the behavior of nearby vehicles. The algorithms presented here for track fusion and behavioral reasoning based on curvilinear coordinates have been verified through experiments in various real traffic scenarios.