Vision Sensor Fusion Research Articles

Advancements in Autonomous Mobile Robot: A Holistic Review of Obstacle Avoidance Methods

The emergence of AMRs has altered our perspective and relationship with automation. At the heart of this transition is navigation and obstacle avoidance, both of which are important needs for deploying AMRs in a variety of scenarios. This comprehensive review looks at the latest advances in navigation and collision avoidance for AMRs, including a wide range of modern techniques and methodologies, algorithms, and technologies that aim to improve functionality. The study provides a detailed analysis of known approaches, such as rule-based approaches, potential fields, reactive navigation systems as behavior systems, and path-following algorithms, that have been developed to address the difficulty in practice. In contrast, technological advancements in machine learning, computer vision sensor fusion, and SLAM techniques, as well as edge computing, are reviewed in light of their unprecedented impact on AMR navigation. Global and local techniques are tackled using universal worldwide optics as well as national adaptations that reveal the unique characteristics of individual countries. The Data Analysis and Processing section emphasizes the importance of technologies that define AMR performance. Due to the constraints imposed by previous studies, it is clear that additional research is required to focus on closing gaps in controlled environments and using standard benchmarks; sensor heterogeneity issues; and practical implementation of theoretical aspects. In a nutshell, this review provides a map of the complex world of AMR navigation and obstacle avoidance. Its primary purpose is to contribute to the continuing debate, promote innovation, and suggest new research avenues in a fast-changing world of autonomous mobile robotics that breaks down traditional deployment constraints.

Towards Sustainable Safe Driving: A Multimodal Fusion Method for Risk Level Recognition in Distracted Driving Status

Precise driving status recognition is a prerequisite for human–vehicle collaborative driving systems towards sustainable road safety. In this study, a simulated driving platform was built to capture multimodal information simultaneously, including vision-modal data representing driver behaviour and sensor-modal data representing vehicle motion. Multisource data are used to quantify the risk of distracted driving status from four levels, safe driving, slight risk, moderate risk, and severe risk, rather than detecting action categories. A multimodal fusion method called vision-sensor fusion transformer (V-SFT) was proposed to incorporate the vision-modal of driver behaviour and sensor-modal data of vehicle motion. Feature concatenation was employed to aggregate representations of different modalities. Then, successive internal interactions were performed to consider the spatiotemporal dependency. Finally, the representations were clipped and mapped into four risk level label spaces. The proposed approach was evaluated under different modality inputs on the collected datasets and compared with some baseline methods. The results showed that V-SFT achieved the best performance with an recognition accuracy of 92.0%. It also indicates that fusing multimodal information effectively improves driving status understanding, and V-SFT extensibility is conducive to integrating more modal data.

Visual Sensor Fusion Based Autonomous Robotic System for Assistive Drinking.

People with severe motor impairments like tetraplegia are restricted in activities of daily living (ADL) and are dependent on continuous human assistance. Assistive robots perform physical tasks in the context of ADLs to support people in need of assistance. In this work a sensor fusion algorithm and a robot control algorithm for localizing the user’s mouth and autonomously navigating a robot arm are proposed for the assistive drinking task. The sensor fusion algorithm is implemented in a visual tracking system which consists of a 2-D camera and a single point time-of-flight distance sensor. The sensor fusion algorithm utilizes computer vision to combine camera images and distance measurements to achieve reliable localization of the user’s mouth. The robot control algorithm uses visual servoing to navigate a robot-handled drinking cup to the mouth and establish physical contact with the lips. This system features an abort command that is triggered by turning the head and unambiguous tracking of multiple faces which enable safe human robot interaction. A study with nine able-bodied test subjects shows that the proposed system reliably localizes the mouth and is able to autonomously navigate the cup to establish physical contact with the mouth.

Sensor Fusion of Vision, Kinetics, and Kinematics for Forward Prediction During Walking With a Transfemoral Prosthesis

Powered lower limb prostheses utilize activity specific controllers, where predicting desired activity and transitioning in a timely manner is essential for seamless locomotion without trips. Previously research considered EMG and mechanical sensors for these predictions; however, predictability must be improved to ensure safe usage. Previously we showed that combining features from mechanical sensors, EMG and Vision yielded greater repeatability, greater separability and lower variability. Here we compare performance of offline forward prediction systems combining these different sensor modalities. We trained and tested subject-specific classifiers for steady-state and transition steps, with data from 8 able-bodied subjects, 4 able-bodied subjects walking with a powered knee-ankle prosthesis using a bypass socket, and a single transfemoral amputee walking with a knee-ankle prosthesis. Fusing Mechanical, EMG, and Vision features produced the best classification for all subjects, with transition error rates in the range of 1% and steady-state error rates close to 0%. Though generalizability was good across able-bodied subjects, it was poor when training with able-bodied or bypass subjects and testing with our amputee subject, regardless of sensor modality, particularly for transition steps. Therefore, we believe a general classifier will require inclusion of amputee training data. Future work will test more subjects and continue development of a general control system.

Aerial Object Detection and Tracking based on Fusion of Vision and Lidar Sensors using Kalman Filter for UAV

Aerial Object Detection and Tracking based on Fusion of Vision and Lidar Sensors using Kalman Filter for UAV

Computer-aided parametric prosthetic socket design based on real-time soft tissue deformation and an inverse approach

The prosthetic socket provides the critical interface between the prosthetic device and the patient’s residual limb. Since each stump is unique in terms of morphology and mechanics, each socket should be patient specific. Computer-aided design solutions have been proposed in the literature. However, there is a lack of an efficient solution able to modify local information based on soft tissue deformation feedback to enhance the design process. The objective of the present work was to develop and evaluate a computer-aided design approach with real-time soft tissue deformation feedback and an inverse approach to optimize the stump–socket interaction. A computer-aided parametric socket design workflow was proposed. Soft tissue deformation was performed using a novel formulation of the mass-spring system. An inverse approach was proposed to estimate and optimize the stump–socket interaction. An interactive parametric design tool was also developed and evaluated. The proposed approach was applied on a CT-based dataset. Finally, the obtained design outcomes were compared with FE simulation outcomes for evaluation purpose. As results, a virtual socket prototype of the CT-based stump model was designed and illustrated by an interactive process. The comparison of stump–socket interaction behavior with FE simulation outcomes showed a very good agreement with a pressure absolute deviation error ranging from 1.44 ± 2.13 to 3.66 ± 4.56 kPa. Moreover, the contact pressures are below the pain-threshold curves, confirming the comfortability of the designed sockets according to the predefined criteria. The present study proposed a computer-aided socket design solution to locally enhance the socket geometry and mechanics. This opens new avenues to increase the design accuracy, reduce the design cost and give the involved patient a geometrically and mechanically fitted socket device. As perspectives, this process will be integrated with our available visual sensor fusion toward a complete computer-aided socket design system for lower limb prosthetic design and fabrication.

Modeling Two-Person Segmentation and Locomotion for Stereoscopic Action Identification: A Sustainable Video Surveillance System

Due to the constantly increasing demand for automatic tracking and recognition systems, there is a need for more proficient, intelligent and sustainable human activity tracking. The main purpose of this study is to develop an accurate and sustainable human action tracking system that is capable of error-free identification of human movements irrespective of the environment in which those actions are performed. Therefore, in this paper we propose a stereoscopic Human Action Recognition (HAR) system based on the fusion of RGB (red, green, blue) and depth sensors. These sensors give an extra depth of information which enables the three-dimensional (3D) tracking of each and every movement performed by humans. Human actions are tracked according to four features, namely, (1) geodesic distance; (2) 3D Cartesian-plane features; (3) joints Motion Capture (MOCAP) features and (4) way-points trajectory generation. In order to represent these features in an optimized form, Particle Swarm Optimization (PSO) is applied. After optimization, a neuro-fuzzy classifier is used for classification and recognition. Extensive experimentation is performed on three challenging datasets: A Nanyang Technological University (NTU) RGB+D dataset; a UoL (University of Lincoln) 3D social activity dataset and a Collective Activity Dataset (CAD). Evaluation experiments on the proposed system proved that a fusion of vision sensors along with our unique features is an efficient approach towards developing a robust HAR system, having achieved a mean accuracy of 93.5% with the NTU RGB+D dataset, 92.2% with the UoL dataset and 89.6% with the Collective Activity dataset. The developed system can play a significant role in many computer vision-based applications, such as intelligent homes, offices and hospitals, and surveillance systems.

End‐Effector Pose Estimation in Complex Environments Using Complementary Enhancement and Adaptive Fusion of Multisensor

Redundant manipulators are suitable for working in narrow and complex environments due to their flexibility. However, a large number of joints and long slender links make it hard to obtain the accurate end‐effector pose of the redundant manipulator directly through the encoders. In this paper, a pose estimation method is proposed with the fusion of vision sensors, inertial sensors, and encoders. Firstly, according to the complementary characteristics of each measurement unit in the sensors, the original data is corrected and enhanced. Furthermore, an improved Kalman filter (KF) algorithm is adopted for data fusion by establishing the nonlinear motion prediction of the end‐effector and the synchronization update model of the multirate sensors. Finally, the radial basis function (RBF) neural network is used to adaptively adjust the fusion parameters. It is verified in experiments that the proposed method achieves better performances on estimation error and update frequency than the original extended Kalman filter (EKF) and unscented Kalman filter (UKF) algorithm, especially in complex environments.

Singing and Nervous System Regulation Based on Wireless Sensor Network Perception

In order to build an intelligent platform that can be applied to singing and nervous system adjustment, this paper optimizes the positioning and information processing algorithms for wireless sensor network perception. Moreover, this article combines binocular vision to realize the singer’s real‐time positioning, combines the singer’s emotion recognition with the intelligent sensor system, and combines the emotion recognition with the adjustment of the nervous system, so that the singer can better control the intelligent platform. In addition, in order to solve the problem of multisensor information fusion, this paper improves the sensor fusion algorithm to make it suitable for the information fusion of vision sensors and information sensors. Finally, this paper designs the functional structure of the system, transmits data through wireless sensor networks, simulates human emotion models, and studies the process of singing and the adjustment of the nervous system. It can be seen from the experimental research results that the method proposed in this paper has a certain effect.

Visual Sensor Fusion With Error Compensation Strategy Toward a Rapid and Low-Cost 3D Scanning System for the Lower Residual Limb

Rapid and low-cost 3D scanning system becomes a common solution for prosthetic designers and engineers to acquire accurate 3D geometrical model of the involved patient. Multiple Kinect sensor fusion solutions have been proposed but still, there is a lack of fusion scheme within a single workstation and a systematic reconstruction error analysis. The objective of the present study was to develop a new Kinect sensor fusion scheme running on a single workstation and an error compensation strategy. Architecture of a 3D scanning system for the lower residual limb was proposed. Four Kinect sensors were used to develop the fusion scheme. An error compensation approach based on linear regression method was performed. Then, an iterative rigid multi-set registration method was proposed. Different primitive geometries (cylindrical objects, box) with well-known physical dimensions were designed, fabricated and used for the evaluation process. Obtained results showed that the error compensation strategy leads to a reduction of around 40% of 3D geometrical reconstruction error for the residual stump-like structure. The total scanning and processing time is 25.4 ± 2.5 seconds. Moreover, our novel iterative rigid multi-set registration method leads to an improvement of accuracy of 60% compared to conventional registration method. Our proposed 3D scanning system using multiple Kinect cameras with error compensation strategy is expected to reduce the design cost and improve the design accuracy. As perspective, this 3D scanning system with fusion scheme will be integrated into a Computer-Aided Design (CAD) system with soft tissue deformation feedbacks for lower limb prostheses.

Detection and Tracking of Road Barrier Based on Radar and Vision Sensor Fusion

The detection and tracking algorithms of road barrier including tunnel and guardrail are proposed to enhance performance and reliability for driver assistance systems. Although the road barrier is one of the key features to determine a safe drivable area, it may be recognized incorrectly due to performance degradation of commercial sensors such as radar and monocular camera. Two frequent cases among many challenging problems are considered with the commercial sensors. The first case is that few tracks of radar to road barrier are detected due to material type of road barrier. The second one is inaccuracy of relative lateral position by radar, thus resulting in large variance of distance between a vehicle and road barrier. To overcome the problems, the detection and estimation algorithms of tracks corresponding to road barrier are proposed. Then, the tracking algorithm based on a probabilistic data association filter (PDAF) is used to reduce variation of lateral distance between vehicle and road barrier. Finally, the proposed algorithms are validated via field test data and their performance is compared with that of road barrier measured by lidar.

Visible and Infrared Video Fusion Using Uniform Discrete Curvelet Transform and Spatial‐Temporal Information

Multiple visual sensor fusion provides aneffective way to improve the robustness and accuracy ofvideo surveillance system. Traditional video fusion methods fuse the source videos using static image fusion methods frame-by-frame without considering the informationin temporal dimension. The temporal information can’t befully utilized in fusion procedure. Aiming at this problem, avisible and infrared video fusion method based on Uniformdiscrete curvelet transform (UDCT) and spatial-temporalinformation is proposed. The source videos are decomposedby using UDCT, and a set of local spatial-temporal energybased fusion rules are designed for decomposition coefficients. In these rules, we consider the current frame’s coefficients and the coefficients on temporal dimension whichare the coefficients of adjacent frames. Experimental results demonstrated that the proposed method works welland outperforms comparison methods in terms of temporalstability and consistency as well as spatial-temporal information extraction.

차선 변경 지원을 위한 레이더 및 비전센서 융합기반 다중 차량 인식

This paper presents a multiple vehicle recognition algorithm based on radar and vision sensor fusion for lane change assistance. To determine whether the lane change is possible, it is necessary to recognize not only a primary vehicle which is located in-lane, but also other adjacent vehicles in the left and/or right lanes. With the given sensor configuration, two challenging problems are considered. One is that the guardrail detected by the front radar might be recognized as a left or right vehicle due to its genetic characteristics. This problem can be solved by a guardrail recognition algorithm based on motion and shape attributes. The other problem is that the recognition of rear vehicles in the left or right lanes might be wrong, especially on curved roads due to the low accuracy of the lateral position measured by rear radars, as well as due to a lack of knowledge of road curvature in the backward direction. In order to solve this problem, it is proposed that the road curvature measured by the front vision sensor is used to derive the road curvature toward the rear direction. Finally, the proposed algorithm for multiple vehicle recognition is validated via field test data on real roads.

다중주기 칼만 필터를 이용한 비동기 센서 융합

We propose a multi-rate sensor fusion of vision and radar using Kalman filter to solve problems of asynchronized and multi-rate sampling periods in object vehicle tracking. A model based prediction of object vehicles is performed with a decentralized multi-rate Kalman filter for each sensor (vision and radar sensors.) To obtain the improvement in the performance of position prediction, different weighting is applied to each sensor’s predicted object position from the multi-rate Kalman filter. The proposed method can provide estimated position of the object vehicles at every sampling time of ECU. The Mahalanobis distance is used to make correspondence among the measured and predicted objects. Through the experimental results, we validate that the post-processed fusion data give us improved tracking performance. The proposed method obtained two times improvement in the object tracking performance compared to single sensor method (camera or radar sensor) in the view point of roots mean square error.

Visual sensor fusion for active security in robotic industrial environments

This work presents a method of information fusion involving data captured by both a standard charge-coupled device (CCD) camera and a time-of-flight (ToF) camera to be used in the detection of the proximity between a manipulator robot and a human. Both cameras are assumed to be located above the work area of an industrial robot. The fusion of colour images and time-of-flight information makes it possible to know the 3D localization of objects with respect to a world coordinate system. At the same time, this allows to know their colour information. Considering that ToF information given by the range camera contains innacuracies including distance error, border error, and pixel saturation, some corrections over the ToF information are proposed and developed to improve the results. The proposed fusion method uses the calibration parameters of both cameras to reproject 3D ToF points, expressed in a common coordinate system for both cameras and a robot arm, in 2D colour images. In addition to this, using the 3D information, the motion detection in a robot industrial environment is achieved, and the fusion of information is applied to the foreground objects previously detected. This combination of information results in a matrix that links colour and 3D information, giving the possibility of characterising the object by its colour in addition to its 3D localisation. Further development of these methods will make it possible to identify objects and their position in the real world and to use this information to prevent possible collisions between the robot and such objects.

6DOF Wireless Tracking Wand Using MARG and Vision Sensor Fusion

We present a low cost battery-powered 6-degree-of-freedom wireless wand for 3D modeling in free space by tri-axis Magnetic, Angular Rate, Gravity (MARG) and vision sensor fusion. Our approach has two stages of sensor fusion, each with different algorithms for finding 3D orientation and position. The first stage fusion algorithm, a complementary filter, utilizes MARG sensors to compute 3D orientation relative to the direction of gravity and earth's magnetic field in a quaternion format, which was adjusted with compensations for magnetic distortion. The second stage fusion algorithm, a Kalman filter, utilizes accelerometer data and IR marker velocity to compute 3D position. In order to compute the IR marker linear velocity along the optical axis (the z-axis), we present a simple and efficient image-based technique to find the distance of the object from the camera using blob area pixels in the image. Our fusion (inside-in and outside-in) approach efficiently solves short time occlusion, needs of frequent calibration, and unbounded drift problems involved in numerical integration of inertial sensors data and improves the degrees of freedom at low cost without compromising accuracy. The results are compared with a leading commercial magnetic motion tracking system to demonstrate the performance of the wand.

Solar Tracking System Experimental Verification Based on GPS and Vision Sensor Fusion

Solar Tracking System Experimental Verification Based on GPS and Vision Sensor Fusion

Measuring indoor occupancy in intelligent buildings using the fusion of vision sensors

In intelligent buildings, practical sensing systems designed to gather indoor occupancy information play an indispensable role in improving occupant comfort and energy efficiency. In this paper, we propose a novel method for occupancy measurement based on the video surveillance now widely used in buildings. In our method, we analyze occupant detection both at the entrance and inside the room. A two-stage static detector is presented based on both appearances and shapes to find the human heads in rooms, and motion-based technology is used for occupant detection at the entrance. To model the change of occupancy and combine the detection results from multiple vision sensors located at entrances and inside rooms for more accurate occupancy estimation, we propose a dynamic Bayesian network-based method. The detection results of each vision sensor play the role of evidence nodes of this network, and thus, we can estimate the true occupancy at time t using the evidence prior to (and including) time t. Experimental results demonstrate the effectiveness and efficiency of the proposed method.

Advances and perspective on motion detection fusion in visual and thermal framework

Multiple visual sensor fusion provides an effective way to improve the robustness and accuracy of visual surveillance system.This paper gives a review of moving object detection with visual-thermal fusion.Firstly,the problem is analyzed,including its procedure,key problems,advantages,testing data sets.Then the existing algorithms are analyzed according to several different rules: fusion strategy,detection method and fusion structure.By comparing different kinds of methods,we analyzed the advantage,disadvantage and applicability.Finally we conclude with discussion on difficulties,challenges and possible future research directions of this kind of method.

2P1-G09 視覚と力覚のセンサフュージョンによるロボットの移動対象物への力制御(作業をするロボット)

2P1-G09 視覚と力覚のセンサフュージョンによるロボットの移動対象物への力制御(作業をするロボット)