Vision-based Sensor Research Articles

Embedded Sensors, Communication Technologies, Computing Platforms and Machine Learning for UAVs: A Review

Unmanned aerial vehicles (UAVs) are increasingly becoming popular due to their use in many commercial and military applications, and their affordability. The UAVs are equipped with various sensors, hardware platforms and software technologies which enable them to support the diverse application portfolio. Sensors include vision-based sensors such as RGB-D cameras, thermal cameras, light detection and ranging (LiDAR), mmWave radars, ultrasonic sensors, and an inertial measurement unit (IMU) which enable UAVs for autonomous navigation, obstacle detection, collision avoidance, object tracking and aerial inspection. To enable smooth operation, UAVs utilize a number of communication technologies such as wireless fidelity (Wi-Fi), long range (LoRa), long-term evolution for machine-type communication (LTE-M), etc., along with various machine learning algorithms. However, each of these different technologies come with their own set of advantages and challenges. Hence, it is essential to have an overview of the different type of sensors, computing and communication modules and algorithms used for UAVs. This paper provides a comprehensive review on the state-of-the-art embedded sensors, communication technologies, computing platforms and machine learning techniques used in autonomous UAVs. The key performance metrics along with operating principles and a detailed comparative study of the various technologies are also studied and presented. The information gathered in this paper aims to serve as a practical reference guide for designing smart sensing applications, low-latency and energy efficient communication strategies, power efficient computing modules and machine learning algorithms for autonomous UAVs. Finally, some of the open issues and challenges for future research and development are also discussed.

A soft thumb-sized vision-based sensor with accurate all-round force perception

Vision-based haptic sensors have emerged as a promising approach to robotic touch due to affordable high-resolution cameras and successful computer vision techniques; however, their physical design and the information they provide do not yet meet the requirements of real applications. We present a robust, soft, low-cost, vision-based, thumb-sized three-dimensional haptic sensor named Insight, which continually provides a directional force-distribution map over its entire conical sensing surface. Constructed around an internal monocular camera, the sensor has only a single layer of elastomer over-moulded on a stiff frame to guarantee sensitivity, robustness and soft contact. Furthermore, Insight uniquely combines photometric stereo and structured light using a collimator to detect the three-dimensional deformation of its easily replaceable flexible outer shell. The force information is inferred by a deep neural network that maps images to the spatial distribution of three-dimensional contact force (normal and shear). Insight has an overall spatial resolution of 0.4 mm, a force magnitude accuracy of around 0.03 N and a force direction accuracy of around five degrees over a range of 0.03–2 N for numerous distinct contacts with varying contact area. The presented hardware and software design concepts can be transferred to a wide variety of robot parts.

Internal Works Quality Assessment for Wall Evenness using Vision-based Sensor on a Mecanum-Wheeled Mobile Robot

Internal Works Quality Assessment for Wall Evenness using Vision-based Sensor on a Mecanum-Wheeled Mobile Robot

Temporal Point Cloud Fusion With Scene Flow for Robust 3D Object Tracking

Non-visual range sensors such as Lidar have shown the potential to detect, locate and track objects in complex dynamic scenes thanks to their higher stability in comparison with vision-based sensors like cameras. However, due to the disorder, sparsity, and irregularity of the point cloud, it is much more challenging to take advantage of the temporal information in the dynamic 3D point cloud sequences, as it has been done in the image sequences for improving detection and tracking. In this paper, we propose a novel scene-flow-based point cloud feature fusion module to tackle this challenge, based on which a 3D object tracking framework is also achieved to exploit the temporal motion information. Moreover, we carefully designed several training schemes that contribute to the success of this new module by eliminating the issues of overfitting and long-tailed distribution of object categories. Extensive experiments on the public KITTI 3D object tracking dataset demonstrate the effectiveness of the proposed method by achieving superior results to the baselines. The source code is available at <uri>https://github.com/Tsinghua-OpenICV/SharingVan-OpenPCDet</uri>.

Bearing Sensor for 3D Localization and Swarm Centroid Detection

A bearing sensor measures its orientation relative to a landmark. Such sensors enable tracking and localization. Since small Unmanned Aerial Vehicles (UAVs) suffer from payload limitations, they must be equipped with lightweight sensors with low power consumption. This paper presents two implementations for bearing sensors. First, a novel vision-based sensor based on a camera and a spherical mirror that enables wide view. This sensor is lightweight and low cost ( 20 [g] and under 8 [USD]). Second, a novel, laser-based beacon together with a photo-diode based sensor which enables to better distinguish between multiple landmarks while keeping inclination mismatch robustness. We present the scheme for detecting the relative angular position of an aerial agent to a light-based landmark in 3D. We also present algorithms for the localization of a UAV, and swarm centroid detection, both using such bearing observations. Finally, we demonstrate our sensors and algorithms in real-world experiments and provide their accuracy.

Underexposed Vision-Based Sensors’ Image Enhancement for Feature Identification in Close-Range Photogrammetry and Structural Health Monitoring

This paper describes an alternative structural health monitoring (SHM) framework for low-light settings or dark environments using underexposed images from vision-based sensors based on the practical implementation of image enhancement algorithms. The proposed framework was validated by two experimental works monitored by two vision systems under ambient lights without assistance from additional lightings. The first experiment monitored six artificial templates attached to a sliding bar that was displaced by a standard one-inch steel block. The effect of image enhancement in the feature identification and bundle adjustment integrated into the close-range photogrammetry were evaluated. The second validation was from a seismic shake table test of a full-scale three-story building tested at E-Defense in Japan. Overall, this study demonstrated the efficiency and robustness of the proposed image enhancement framework in (i) modifying the original image characteristics so the feature identification algorithm is capable of accurately detecting, locating and registering the existing features on the object; (ii) integrating the identified features into the automatic bundle adjustment in the close-range photogrammetry process; and (iii) assessing the measurement of identified features in static and dynamic SHM, and in structural system identification, with high accuracy.

Efficient Video-based Vehicle Queue Length Estimation using Computer Vision and Deep Learning for an Urban Traffic Scenario

In the Intelligent Transportation System (ITS) realm, queue length estimation is one of an essential yet a challenging task. Queue lengths are important for determining traffic density in traffic lanes so that possible congestion in any lane can be minimized. Smart roadside sensors such as loop detectors, radars and pneumatic road tubes etc. are promising for such tasks though they have a very high installation and maintenance cost. Large scale deployment of surveillance cameras have shown a great potential in the collection of vehicular data in a flexible way and are also cost effective. Similarly, vision-based sensors can be used independently or if required can also augment the functionality of other roadside sensors to effectively process queue length at prescribed traffic lanes. In this research, a CNN-based approach for estimation of vehicle queue length in an urban traffic scenario using low-resolution traffic videos is proposed. The queue length is estimated based on count of total vehicles waiting on a signal. The proposed approach calculates queue length without the knowledge of any onsite camera calibration information. Average vehicle length is approximated to be 5 m. This caters for the vehicles at the far end of the traffic lane that appear smaller in the camera view. Identification of stopped vehicles is done using Deep SORT based object tracking. Due to robust and accurate CNN-based detection and tracking, the queue length estimated by using only the cameras has been very effective. This mostly eliminates the need for fusion with any roadside or in-vehicle sensors. A detailed comparative analysis of vehicle detection models including YOLOv3, YOLOv4, YOLOv5, SSD, ResNet101, and InceptionV3 was performed. Based on this analysis, YOLOv4 was selected as a baseline model for queue length estimation. Using the pre-trained 80-classes YOLOv4 model, an overall accuracy of 73% and 88% was achieved for vehicle count and vehicle count-based queue length estimation, respectively. After fine-tuning of model and narrowing the output classes to vehicle class only, an average accuracy of 83% and 93% was achieved, respectively. This shows the efficiency and robustness of the proposed approach.

An Elderly Monitoring System Based on Multiple Ultra-Sensitive Vibration and Pneumatic Sensors

In recent years, aging of the population has become a major social problem. Various types of sensor networks have been applied to elderly monitoring for addressing the care problems of the elderly living alone. We have also proposed elderly monitoring systems based on wireless sensor network devices. However, vision-based sensors can also cause a mental burden on the elderly’s privacy. Furthermore, the number of sensors must be reduced, if possible. Therefore, this study proposes an elderly monitoring system composed of two vibration sensors placed on the floor and a pneumatic sensor placed on the bed. Because both sensors include considerable measurement noise, we propose a human behavior estimation method that includes anomaly detection from time-series measurement data using an autocorrelation coefficient. Finally, we discuss the effectiveness and usability of the proposed system through several experimental results. The accuracy of walking detection reaches 94.4%, while the error of heartbeat detection is 3.01 bpm.

Robust Multisensor Fusion for the Development of EEG Controlled Vehicle

Stroke survivors or patients suffering from paralyses face a lot of challenges when it comes to mobility. Autonomous vehicle technologies can help to increase the accessibility for persons having limited mobility. The design of such vehicle requires a network of sensors along with sensor fusion system to accurately map the surrounding environment necessary for its safe driving. This study proposes a brain-actuated intelligent vehicle with network of sonars and vision-based sensors which can be controlled by either using Steady-State Visual Evoked Potential brain signals or joystick. For electroencephalogram signal detection, four frequencies were flickered and displayed using a GUI on LCD screen and signals were recorded having 1000 Hz sampling frequency. These signals were segmented into 2-second window size with 50% overlap and features were extracted by using canonical correlation analysis. In addition, a low-level navigation control system consisting of smart sensor network was also implemented to maximize safety of the user. Data from various sensors was sent to the intelligent control unit for sensor fusion in order to improve the performance and robustness of the designed vehicle. The results were evaluated for both offline and online classification. For offline control, the accuracy of 96% was achieved with 104.48 bits/min while 95% accuracy was achieved for run-time control. The results show that the proposed system has improved accuracy as well as information transfer rate is concerned as compared to the existing systems.

RGB-D Data-Based Action Recognition: A Review.

Classification of human actions is an ongoing research problem in computer vision. This review is aimed to scope current literature on data fusion and action recognition techniques and to identify gaps and future research direction. Success in producing cost-effective and portable vision-based sensors has dramatically increased the number and size of datasets. The increase in the number of action recognition datasets intersects with advances in deep learning architectures and computational support, both of which offer significant research opportunities. Naturally, each action-data modality—such as RGB, depth, skeleton, and infrared (IR)—has distinct characteristics; therefore, it is important to exploit the value of each modality for better action recognition. In this paper, we focus solely on data fusion and recognition techniques in the context of vision with an RGB-D perspective. We conclude by discussing research challenges, emerging trends, and possible future research directions.

Semantic Evidential Grid Mapping Using Monocular and Stereo Cameras.

Accurately estimating the current state of local traffic scenes is one of the key problems in the development of software components for automated vehicles. In addition to details on free space and drivability, static and dynamic traffic participants and information on the semantics may also be included in the desired representation. Multi-layer grid maps allow the inclusion of all of this information in a common representation. However, most existing grid mapping approaches only process range sensor measurements such as Lidar and Radar and solely model occupancy without semantic states. In order to add sensor redundancy and diversity, it is desired to add vision-based sensor setups in a common grid map representation. In this work, we present a semantic evidential grid mapping pipeline, including estimates for eight semantic classes, that is designed for straightforward fusion with range sensor data. Unlike other publications, our representation explicitly models uncertainties in the evidential model. We present results of our grid mapping pipeline based on a monocular vision setup and a stereo vision setup. Our mapping results are accurate and dense mapping due to the incorporation of a disparity- or depth-based ground surface estimation in the inverse perspective mapping. We conclude this paper by providing a detailed quantitative evaluation based on real traffic scenarios in the KITTI odometry benchmark dataset and demonstrating the advantages compared to other semantic grid mapping approaches.

Vision-based modal parameter identification for bridges using a novel holographic visual sensor

Noncontact monitoring using holography can be quite useful in vision-based structural health monitoring owing to its advantages in terms of data visualization and real-time applications. Aimed at exploring the potential of vision-based sensors for structural health monitoring in more general applications, this paper has developed the holographic visual sensor (HVS) with a vision-based modal parameter identification method. This strategy is an extension of the current vision-based techniques and holography. Based on the newly introduced spatial and temporal sequence data measured by HVS, the vision-based modal parameter identification method was proposed and applied to structural health monitoring for the first time. Accordingly, the feature point set describing the geometric morphology of the structural holography and the time-state-space mathematical model of the structural dynamics system was constructed. In this way, the mechanical behavior characteristics could be extracted from serial transient holographic responses following excitation, and the structural modal parameters were then calculated and quantitatively analyzed. Laboratory experiments and field tests were employed for validation based on the comparison between the results determined by the proposed HVS and those determined by conventional, high-precision contact sensors. The developed HVS and identification algorithms were able to visually collect and identify modal parameters at a higher data resolution and provide smoother modal shapes. The proposed method can thus be used to complement and improve existing computer-vision-based measurement techniques and damage detection.

Strategy for on-orbit space object classification using deep learning

Since nanosatellites are spotlighted as a verification platform for space technology, new studies on on-orbit satellite servicing using nanosatellites are being conducted. This servicing is based on space robotics using vision-based sensors in the rendezvous state with a target satellite. The space environment, such as sunlight and Earth albedo, affects the mission. Simulation of the space environment on the ground is difficult, but the development of robust algorithms which reflect the effect is essential. In particular, missions such as active debris removal require a method for overcoming changes in any known information due to external factors such as collisions. This study proposes a new strategy on nanosatellite for on-orbit space object classification by applying deep learning to sensor-based orbit satellite service activity. When previously known information is changed, a method of online learning on orbit after obtaining additional data at a short relative distance can help determine the final service part. Using the images and point cloud data that simulate the space environment, we apply a convolutional neural network and PointNet to classify the objects. The learning environment is studied using a general desktop and a micro-graphics processing unit (GPU) board that can be mounted on a nanosatellite. For the training, we used self-produced data using 3D models of nanosatellites and asteroids with similar shapes, which are difficult to distinguish with existing algorithms. Consequently, the proposed strategy by the author shows feasibility of using nanosatellite’s micro GPU for on-orbit space object classification, and it is verified that point cloud–based methods are more suitable by utilizing deep learning for nanosatellites. The proposed method with processor of nanosatellite is applicable to satellite service missions in orbit, such as capturing of robotic parts for extending life span or removing space debris.

Visual sensing technologies in robotic welding: Recent research developments and future interests

Vision-based sensors play an essential role in robotic welding, which attract increasing attention by scholars because of its non-contact characteristic, high precision, fast detection, and strong adaptability. This paper summarizes the recent research developments of vision sensor-based intelligent robotic welding and analyzes their characteristics and application scenarios in the welding process. The applications of vision sensing technologies are illustrated, especially in visual calibration, initial welding position guidance, weld seam tracking, weld pool monitoring, correlation deformation detection, and defect identification. At the end of this paper, the promising development directions of intelligent robotic welding technologies are discussed, including the hardware developments of vision sensors, optimization of image processing algorithms, the combination of passive vision and active vision, multi-sensor information fusion, and applications of artificial intelligence.

Multilayer Mapping Kit for Autonomous UAV Navigation

Mapping, as the back-end of perception and the front-end of path planning in the modern UAV navigation system, draws our interest. Considering the requirements of UAV navigation and the features of the current embedded computation platforms, we designed and implemented a novel multilayer mapping framework. In this framework, we divided the map into three layers: awareness, local, and global. The awareness map is constructed in cylindrical coordinate, enabling fast raycasting. The local map is a probability-based volumetric map. The global map adopts dynamic memory management, allocating memory for the active mapping area, and recycling the memory from the inactive mapping area. We implemented this mapping framework in three parallel threads: awareness thread, local-global thread, and visualization thread. Finally, we evaluated the mapping kit in both the simulation environment and the real-world scenario with the vision-based sensors. The framework supports different kinds of map outputs for the global or local path planners. The implementation is open-source for the research community.

An Extensive Analysis of the Vision-based Deep Learning Techniques for Action Recognition

Action recognition involves the idea of localizing and classifying actions in a video over a sequence of frames. It can be thought of as an image classification task extended temporally. The information obtained over the multitude of frames is aggregated to comprehend the action classification output. Applications of action recognition systems range from assistance for healthcare systems to human-machine interaction. Action recognition has proven to be a challenging task as it poses many impediments including high computation cost, capturing extended context, designing complex architectures, and lack of benchmark datasets. Increasing the efficiency of algorithms in human action recognition can significantly improve the probability of implementing it in real-world scenarios. This paper has summarized the evolution of various action localization, classification, and detection algorithms applied to data from vision-based sensors. We have also reviewed the datasets that have been used for the action classification, localization, and detection process. We have further explored the areas of action classification, temporal and spatiotemporal action detection, which use convolution neural networks, recurrent neural networks, or a combination of both.

In-hand manipulation by a robot hand equipped with friction-variable surface and vision-based tactile sensor

In-hand manipulation by a robot hand equipped with friction-variable surface and vision-based tactile sensor

MmFall: Fall Detection Using 4-D mmWave Radar and a Hybrid Variational RNN AutoEncoder

Elderly fall prevention and detection becomes extremely crucial with the fast aging population globally. In this article, we propose <i>mmFall</i>, a novel fall detection system, which comprises 1) the emerging millimeter-wave (mmWave) radar sensor to collect the human body&#x2019;s point cloud along with the body centroid and 2) a hybrid variational recurrent neural network (RNN) autoencoder (HVRAE) to compute the anomaly level of the body motion based on the acquired point cloud. A fall is detected when the spike in anomaly level and the drop in centroid height occur simultaneously. The mmWave radar sensor offers privacy-compliance and high sensitivity to motion, over the traditional sensing modalities. However, 1) randomness in radar point cloud and 2) difficulties in fall collection/labeling in the traditional supervised fall detection approaches are the two major challenges. To overcome the randomness in radar data, the proposed HVRAE uses variational inference, a generative approach rather than a discriminative approach, to infer the posterior probability of the body&#x2019;s latent motion state every frame, followed by a RNN to summarize the temporal features over multiple frames. Moreover, to circumvent the difficulties in fall data collection/labeling, the HVRAE is built upon an autoencoder architecture in a semisupervised approach, which is only trained on the normal activities of daily living (ADL). In the inference stage, the HVRAE will generate a spike in the anomaly level once an abnormal motion, such as fall, occurs. During the experiment,<xref ref-type="fn" rid="fn1">¹</xref> we implemented the HVRAE along with two other baselines, and tested on the data set collected in an apartment. The receiver operating characteristic (ROC) curve indicates that our proposed model outperforms baselines and achieves 98&#x0025; detection out of 50 falls at the expense of just 2 false alarms. <i>Note to Practitioners</i>&#x2014;Traditional nonwearable fall detection approaches typically make use of a vision-based sensor, such as camera, to monitor and detect fall using a classifier that is trained in a supervised fashion on the collected fall and nonfall data. However, several problems render these methods impractical. First, camera-based monitoring may trigger privacy concerns. Second, fall data collection using human subjects is difficult and costly, not to mention the impossible ask of the elderly repeating simulated falls for data collection. In this article, we propose a new fall detection approach to overcome these problems 1) using a palm-size mmWave radar sensor to monitor the elderly, that is highly sensitive to motion while protecting privacy and 2) using a semisupervised anomaly detection approach to circumvent the fall data collection. Further hardware engineering and more training data from people with different body figures could make the proposed fall detection solution even more practical.

Multiple Nodes Access of Wireless Beam Modulation for 6G-Enabled Internet of Things

For the sixth-generation (6G) communication, it is required to support the massive Internet of Things (IoT) devices with high data rate applications such as vision-based sensors. Unfortunately, the current sub-6-GHz band is already congested, and lower-band communication is hard to support the high data rate transmission. The millimeter-wave (mmWave)-based communication technology is a promising solution to support the high rate transmission in a new band. However, the high power consumption and hardware cost of the mmWave communication system violates the low-cost design principle of IoT devices, which is a heavy burden for the mmWave application on IoT. In this work, we propose the multiple IoT nodes access-based wireless beam modulation (WBM) technology for the mmWave transmission, where the WBM employs the over-the-air modulation to achieve the low-cost mmWave hardware design with hundreds of megabit per second transmission. To support multiple IoT nodes access, we first design a hyper grant-based and grant-free protocols to achieve the balance between robustness and low latency access. The multiple beam search method is proposed to accelerate the beam alignment process. The channel estimation of WBM is achieved by the pilot detection and beam estimation process, where an iterative pilot cancelation beam estimation method is proposed. According to the simulation and hardware implementation results, the WBM-based communication system can provide a much higher data rate transmission service than the current IoT-based communication systems, and much lower power consumption and hardware cost than the mmWave-based system.

Pole-Like Object Extraction and Pole-Aided GNSS/IMU/LiDAR-SLAM System in Urban Area.

Vision-based sensors such as LiDAR (Light Detection and Ranging) are adopted in the SLAM (Simultaneous Localization and Mapping) system. In the 16-beam LiDAR aided SLAM system, due to the difficulty of object detection by sparse laser data, neither the grid-based nor feature point-based solution can avoid the interference of moving objects. In an urban environment, the pole-like objects are common, invariant and have distinguishing characteristics. Therefore, it is suitable to bring more robust and reliable positioning results as auxiliary information in the process of vehicle positioning and navigation. In this work, we proposed a scheme of a SLAM system using a GNSS (Global Navigation Satellite System), IMU (Inertial Measurement Unit) and LiDAR sensor using the position of pole-like objects as the features for SLAM. The scheme combines a traditional preprocessing method and a small scale artificial neural network to extract the pole-like objects in environment. Firstly, the threshold-based method is used to extract the pole-like object candidates from the point cloud, and then, the neural network is applied for training and inference to obtain pole-like objects. The result shows that the accuracy and recall rate are sufficient to provide stable observation for the following SLAM process. After extracting the poles from the LiDAR point cloud, their coordinates are added to the feature map, and the nonlinear optimization of the front end is carried out by utilizing the distance constraints corresponding to the pole coordinates; then, the heading angle and horizontal plane translation are estimated. The ground feature points are used to enhance the elevation, pitch and roll angle accuracy. The performance of the proposed navigation system is evaluated through field experiments by checking the position drift and attitude errors during multiple two-min mimic GNSS outages without additional IMU motion constrain such as NHC (Nonholonomic Constrain). The experimental results show that the performance of the proposed scheme is superior to that of the conventional feature point grid-based SLAM with the same back end, especially in congested crossroads where slow-moving vehicles are surrounded and pole-like objects are rich in the environment. The mean plane position error during two-min GNSS outages was reduced by 38.5%, and the root mean square error was reduced by 35.3%. Therefore, the proposed pole-like feature-based GNSS/IMU/LiDAR SLAM system can fuse condensed information from those sensors effectively to mitigate positioning and orientation errors, even in a short-time GNSS denied environment.