On-board Computer Vision Research Articles

Track geometry monitoring by an on-board computer-vision-based sensor system

This paper illustrates some outcomes of the EU project Assets4Rail, funded by the Shift2Rail Joint Undertaking, as well as on-going research developments.Nowadays, many Track Recording Vehicles (TRV) are equipped with contactless optical/inertial systems to monitor Track Geometry (TG). The dedicated trains and sophisticated measurement equipment are costly to acquire and maintain. Therefore, the interval between two TRV recordings of the TG on the same line section cannot be too short (twice per month to twice per year).Recently, railway operators are showing increasing interest in the use of commercial trains to monitor track condition in a cost-effective manner.TRVs' expensive and constantly maintained optical systems make them unsuitable for commercial fleets. On-board sensor systems based on indirect measurements such as accelerations have been developed in various studies. While detection of the vertical irregularity is relatively straightforward through double-integration of the recorded acceleration, it is yet an unsolved issue for lateral irregularities due to the complicated relative wheel-rail motion.The investigated system conceptually combines on-board measurements of wheel-rail lateral relative position and lateral axlebox acceleration to detect rail alignment variations. It includes a functional prototype of an on-board Computer Vision (CV) sensor capable of monitoring the Lateral Displacement of the Wheel with respect to the Rail (LDWR). Further progress is aiming at the reduction of the measurement errors due to wheelset lateral displacements relative to the track, which is essential for calculating rail alignment.The sensor system prototype was tested in Italy at 100 km/h on the Aldebaran 2.0 TRV of RFI (Rete Ferroviaria Italiana), the main Italian Infrastructure Manager. It was found that the estimated lateral displacement correlates to the lateral alignment acquired by the Aldebaran 2.0 commercial TG inspection equipment.Moreover, a Simpack® simulation of a multi-body model of Aldebaran 2.0 provides the axlebox acceleration values not measurable during the test runs, to evaluate the relationship between them, LDWR and track alignment.

SpaceDrones 2.0—Hardware-in-the-Loop Simulation and Validation for Orbital and Deep Space Computer Vision and Machine Learning Tasking Using Free-Flying Drone Platforms

The proliferation of reusable space vehicles has fundamentally changed how assets are injected into the low earth orbit and beyond, increasing both the reliability and frequency of launches. Consequently, it has led to the rapid development and adoption of new technologies in the aerospace sector, including computer vision (CV), machine learning (ML)/artificial intelligence (AI), and distributed networking. All these technologies are necessary to enable truly autonomous “Human-out-of-the-loop” mission tasking for spaceborne applications as spacecrafts travel further into the solar system and our missions become more ambitious. This paper proposes a novel approach for space-based computer vision sensing and machine learning simulation and validation using synthetically trained models to generate the large amounts of space-based imagery needed to train computer vision models. We also introduce a method of image data augmentation known as domain randomization to enhance machine learning performance in the dynamic domain of spaceborne computer vision to tackle unique space-based challenges such as orientation and lighting variations. These synthetically trained computer vision models then apply that capability for hardware-in-the-loop testing and evaluation via free-flying robotic platforms, thus enabling sensor-based orbital vehicle control, onboard decision making, and mobile manipulation similar to air-bearing table methods. Given the current energy constraints of space vehicles using solar-based power plants, cameras provide an energy-efficient means of situational awareness when compared to active sensing instruments. When coupled with computationally efficient machine learning algorithms and methods, it can enable space systems proficient in classifying, tracking, capturing, and ultimately manipulating objects for orbital/planetary assembly and maintenance (tasks commonly referred to as In-Space Assembly and On-Orbit Servicing). Given the inherent dangers of manned spaceflight/extravehicular activities (EVAs) currently employed to perform spacecraft maintenance and the current limitation of long-duration human spaceflight outside the low earth orbit, space robotics armed with generalized sensing and control and machine learning architecture have a unique automation potential. However, the tools and methodologies required for hardware-in-the-loop simulation, testing, and validation at a large scale and at an affordable price point are in developmental stages. By leveraging a drone’s free-flight maneuvering capability, theater projection technology, synthetically generated orbital and celestial environments, and machine learning, this work strives to build a robust hardware-in-the-loop testing suite. While the focus of the specific computer vision models in this paper is narrowed down to solving visual sensing problems in orbit, this work can very well be extended to solve any problem set that requires a robust onboard computer vision, robotic manipulation, and free-flight capabilities.

Ag-YOLO: A Real-Time Low-Cost Detector for Precise Spraying With Case Study of Palms.

To date, unmanned aerial vehicles (UAVs), commonly known as drones, have been widely used in precision agriculture (PA) for crop monitoring and crop spraying, allowing farmers to increase the efficiency of the farming process, meanwhile reducing environmental impact. However, to spray pesticides effectively and safely to the trees in small fields or rugged environments, such as mountain areas, is still an open question. To bridge this gap, in this study, an onboard computer vision (CV) component for UAVs is developed. The system is low-cost, flexible, and energy-effective. It consists of two parts, the hardware part is an Intel Neural Compute Stick 2 (NCS2), and the software part is an object detection algorithm named the Ag-YOLO. The NCS2 is 18 grams in weight, 1.5 watts in energy consumption, and costs about $66. The proposed model Ag-YOLO is inspired by You Only Look Once (YOLO), trained and tested with aerial images of areca plantations, and shows high accuracy (F1 score = 0.9205) and high speed [36.5 frames per second (fps)] on the target hardware. Compared to YOLOv3-Tiny, Ag-YOLO is 2× faster while using 12× fewer parameters. Based on this study, crop monitoring and crop spraying can be synchronized into one process, so that smart and precise spraying can be performed.

REAL-TIME DEEP NEURAL NETWORKS FOR MULTIPLE OBJECT TRACKING AND SEGMENTATION ON MONOCULAR VIDEO

Abstract. The paper is devoted to the task of multiple objects tracking and segmentation on monocular video, which was obtained by the camera of unmanned ground vehicle. The authors investigate various architectures of deep neural networks for this task solution. Special attention is paid to deep models providing inference in real time. The authors proposed an approach based on combining the modern SOLOv2 instance segmentation model, a neural network model for embedding generation for each found object, and a modified Hungarian tracking algorithm. The Hungarian algorithm was modified taking into account the geometric constraints on the positions of the found objects on the sequence of images. The investigated solution is a development and improvement of the state-of-the-art PointTrack method. The effectiveness of the proposed approach is demonstrated quantitatively and qualitatively on the popular KITTI MOTS dataset collected using the cameras of a driverless car. The software implementation of the approach was carried out. The acceleration of the procedure for the formation of a two-dimensional point cloud in the found image segment was done using the NVidia CUDA technology. At the same time, the proposed instance segmentation module provides a mean processing time of one image of 68 ms, the embedding and tracking module of 24 ms using the NVidia Tesla V100 GPU. This indicates that the proposed solution is promising for on-board computer vision systems for both unmanned vehicles and various robotic platforms.

Drone-Based Ceramic Insulators Condition Monitoring

This article develops a prototype quadcopter drone-based system for inspection of power-line ceramic insulators. The drone uses its onboard cameras and Raspberry Pi single-board computer to monitor the health condition of outdoor ceramic insulators. The main contribution of this article is the development of a complete quadcopter-based system prototype for overhead power-line ceramic insulators inspection. The system is capable of performing the required computer vision routines for insulator health monitoring, either onboard or on an onshore ground station computer. In the onshore mode of operation, the drone captures images as it flies and simultaneously sends them to the onshore ground station. The developed system is tested in real life on a small-scale model frame, on which insulators are mounted. The results presented in this article show that quadcopter-based insulator inspection can be carried out successfully using both onshore and onboard computer vision techniques, with acceptable quality in terms of precision and computer vision time.

Hybrid Electric Vehicle Energy Management With Computer Vision and Deep Reinforcement Learning

Modern automotive systems have been equipped with a highly increasing number of onboard computer vision hardware and software, which are considered to be beneficial for achieving eco-driving. This article combines computer vision and deep reinforcement learning (DRL) to improve the fuel economy of hybrid electric vehicles. The proposed method is capable of autonomously learning the optimal control policy from visual inputs. The state-of-the-art convolutional neural networks-based object detection method is utilized to extract available visual information from onboard cameras. The detected visual information is used as a state input for a continuous DRL model to output energy management strategies. To evaluate the proposed method, we construct 100 km real city and highway driving cycles, in which visual information is incorporated. The results show that the DRL-based system with visual information consumes 4.3–8.8% less fuel compared with the one without visual information, and the proposed method achieves 96.5% fuel economy of the global optimum-dynamic programming.

A Vision-Based Approach for Unmanned Aerial Vehicle Landing

In this paper we present an on-board Computer Vision System for the pose estimation of an Unmanned Aerial Vehicle (UAV) with respect to a human-made landing target. The proposed methodology is based on a coarse-to-fine approach to search the target marks starting from the recognition of the characteristics visible from long distances, up to the inner details when short distances require high precisions for the final landing phase. A sequence of steps, based on a Point-to-Line Distance method, analyzes the contour information and allows the recognition of the target also in cluttered scenarios. The proposed approach enables to fully assist the UAV during its take-off and landing on the target, as it is able to detect anomalous situations, such as the loss of the target from the image field of view, and the precise evaluation of the drone attitude when only a part of the target remains visible in the image plane. Several indoor and outdoor experiments have been carried out to demonstrate the effectiveness, robustness and accuracy of developed algorithm. The outcomes have proven that our methodology outperforms the current state of art, providing high accuracies in estimating the position and the orientation of landing target with respect to the UAV.

Observer design for monocular visual inertial SLAM

Abstract This paper examines the state estimation problem for unmanned aerial vehicles when commonly used positioning systems such as the global positioning system or indoor motion capture systems are unavailable. The proposed method uses inertial sensor measurements along with scaled position measurements from an onboard computer vision system which implements visual simultaneous localization and mapping. A state transformation puts the system into a linear time-varying form which simplifies observability analysis and allows for an observer design with sufficient conditions for convergence. The proposed design is validated by simulation.

Object Detection Method in Traffic by On-Board Computer Vision with Time Delay Neural Network

A collision avoidance system play an important role in reducing incidents occurring on the road, which the object detection is crucial to enable obstacle avoidance for this system. The objective of this paper is to improve general object detection methods for vehicles in order to prevent a collision of the vehicles and the obstacles - of which we do not know the exact shape, size or color. A combined computer vision system with artificial neural networks can improve the performance of the vehicle has the ability to see and recognize the obstacles like human beings. In this paper, the authors present the algorithm for vehicles to detect general objects, which can classify obstacles that are real obstacles or fake obstacles, such as a painting or text on the road. The proposed method, we combined on-board computer vision system based on Histograms of Oriented Gradient (HOG) and Time Delay Neural Network (TDNN). We extract feature of the obstacles by HOG and using TDNN to recognize and classify the obstacles. The experimental results showed that this system can detect general objects, and is not restricted to vehicles, objects or pedestrians. It has provided good results along with high accuracy and reliability, which it is accurate enough to provide a warning to the driver when a collision is imminent.

General Object Detection Method by On-Board Computer Vision with Artificial Neural Networks

General Object Detection Method by On-Board Computer Vision with Artificial Neural Networks

Bioinspired decision architectures containing host and microbiome processing units

Biomimetic robots have been used to explore and explain natural phenomena ranging from the coordination of ants to the locomotion of lizards. Here, we developed a series of decision architectures inspired by the information exchange between a host organism and its microbiome. We first modeled the biochemical exchanges of a population of synthetically engineered E. coli. We then built a physical, differential drive robot that contained an integrated, onboard computer vision system. A relay was established between the simulated population of cells and the robot’s microcontroller. By placing the robot within a target-containing a two-dimensional arena, we explored how different aspects of the simulated cells and the robot’s microcontroller could be integrated to form hybrid decision architectures. We found that distinct decision architectures allow for us to develop models of computation with specific strengths such as runtime efficiency or minimal memory allocation. Taken together, our hybrid decision architectures provide a new strategy for developing bioinspired control systems that integrate both living and nonliving components.

Automated Decision Making in Road Traffic Monitoring by On-Board Unmanned Aerial Vehicle System

The study is dedicated to solving the target issues of the ground traffic monitoring aided by the Unmanned Aerial Vehicles (UAV) based on applying the on-board computer vision systems. It has been shown that making decisions on recognizing the occurring traffic situations under the conflicts of functional criteria is a complicated task, affecting considerably the costs associated with elimination of the consequences of such situations. It has been suggested that the classes of the occurring traffic situations should be identified taking into account the necessity to engage the rapid action team of the rescue service. The models have been proposed describing the functional criteria of losses, of the flight safety of the unmanned aerial vehicle and of the class recognition reliability. The issues related to making decisions on recognizing the occurring traffic situations have been considered. The analysis of the strategies to recognize the situation classes have been carried out based on the principles of minimizing the overall losses, on limiting the admissible UAV flight altitude and on ensuring the required class recognition reliability. It has been shown that applying the minimum loss criterion ensures considerable savings of resources under different ratio of the loss quotients.

Enhanced flyby science with onboard computer vision: Tracking and surface feature detection at small bodies

Spacecraft autonomy is crucial to increase the science return of optical remote sensing observations at distant primitive bodies. To date, most small bodies exploration has involved short timescale flybys that execute prescripted data collection sequences. Light time delay means that the spacecraft must operate completely autonomously without direct control from the ground, but in most cases the physical properties and morphologies of prospective targets are unknown before the flyby. Surface features of interest are highly localized, and successful observations must account for geometry and illumination constraints. Under these circumstances onboard computer vision can improve science yield by responding immediately to collected imagery. It can reacquire bad data or identify features of opportunity for additional targeted measurements. We present a comprehensive framework for onboard computer vision for flyby missions at small bodies. We introduce novel algorithms for target tracking, target segmentation, surface feature detection, and anomaly detection. The performance and generalization power are evaluated in detail using expert annotations on data sets from previous encounters with primitive bodies.

THE PIXHAWK OPEN-SOURCE COMPUTER VISION FRAMEWORK FOR MAVS

Abstract. Unmanned aerial vehicles (UAV) and micro air vehicles (MAV) are already intensively used in geodetic applications. State of the art autonomous systems are however geared towards the application area in safe and obstacle-free altitudes greater than 30 meters. Applications at lower altitudes still require a human pilot. A new application field will be the reconstruction of structures and buildings, including the facades and roofs, with semi-autonomous MAVs. Ongoing research in the MAV robotics field is focusing on enabling this system class to operate at lower altitudes in proximity to nearby obstacles and humans. PIXHAWK is an open source and open hardware toolkit for this purpose. The quadrotor design is optimized for onboard computer vision and can connect up to four cameras to its onboard computer. The validity of the system design is shown with a fully autonomous capture flight along a building.

PIXHAWK: A micro aerial vehicle design for autonomous flight using onboard computer vision

We describe a novel quadrotor Micro Air Vehicle (MAV) system that is designed to use computer vision algorithms within the flight control loop. The main contribution is a MAV system that is able to run both the vision-based flight control and stereo-vision-based obstacle detection parallelly on an embedded computer onboard the MAV. The system design features the integration of a powerful onboard computer and the synchronization of IMU-Vision measurements by hardware timestamping which allows tight integration of IMU measurements into the computer vision pipeline. We evaluate the accuracy of marker-based visual pose estimation for flight control and demonstrate marker-based autonomous flight including obstacle detection using stereo vision. We also show the benefits of our IMU-Vision synchronization for egomotion estimation in additional experiments where we use the synchronized measurements for pose estimation using the 2pt+gravity formulation of the PnP problem.

Hardware‐in‐the‐loop simulation for visual target tracking of octorotor UAV

PurposeThe purpose of this paper is to present the development of hardware‐in‐the‐loop simulation (HILS) for visual target tracking of an octorotor unmanned aerial vehicle (UAV) with onboard computer vision.Design/methodology/approachHILS for visual target tracking of an octorotor UAV is developed by integrating real embedded computer vision hardware and camera to software simulation of the UAV dynamics, flight control and navigation systems run on Simulink. Visualization of the visual target tracking is developed using FlightGear. The computer vision system is used to recognize and track a moving target using feature correlation between captured scene images and object images stored in the database. Features of the captured images are extracted using speed‐up robust feature (SURF) algorithm, and subsequently matched with features extracted from object image using fast library for approximate nearest neighbor (FLANN) algorithm. Kalman filter is applied to predict the position of the moving target on image plane. The integrated HILS environment is developed to allow real‐time testing and evaluation of onboard embedded computer vision for UAV's visual target tracking.FindingsUtilization of HILS is found to be useful in evaluating functionality and performance of the real machine vision software and hardware prior to its operation in a flight test. Integrating computer vision with UAV enables the construction of an unmanned system with the capability of tracking a moving object.Practical implicationsHILS for visual target tracking of UAV described in this paper could be applied in practice to minimize trial and error in various parameters tuning of the machine vision algorithm as well as of the autopilot and navigation system. It also could reduce development costs, in addition to reducing the risk of crashing the UAV in a flight test.Originality/valueA HILS integrated environment for octorotor UAV's visual target tracking for real‐time testing and evaluation of onboard computer vision is proposed. Another contribution involves implementation of SURF, FLANN, and Kalman filter algorithms on an onboard embedded PC and its integration with navigation and flight control systems which enables the UAV to track a moving object.

Autonomously Flying Helicopter

Abstract This paper describes the development of the aerial robotics model helicopter of the Swiss Team ETH Zurich. It also discusses the on-board instrumentation of the helicopter, the realization of the autopilot, as, well as the design of the mission commandar. Additiona1 features like on-board computer vision and a disk retrieval scheme conclude this summary of aerial robotics challenges.

A Preview Steering Autopilot Control Algorithm for Four-Wheel-Steering Passenger Vehicles

This paper addresses the control law design of a preview steering autopilot for a four-wheel-steering vehicle to perform automatic lane tracking. In the concept, an on-board computer vision system is used in lieu of the driver’s vision to track the roadway. The steering autopilot design is formulated as an optimal, discrete-time preview path tracking problem under the “perfect measurement” assumption. Simulation results indicate that the tracking performance of the steering autopilot was improved by preview relative to that calculated for an autopilot without preview. These results also indicate the existence of an effective preview time with which almost all the benefits of previewing future information can be obtained. This effective preview time is about three times the reciprocal of the autopilot’s bandwidth. Our study also indicates that preview steering autopilots can tolerate the use of actuators with a lower bandwidth than those designed without preview information.