On-board Camera Research Articles

Towards safer general aviation operations using a vision-based decision support system for weather threat avoidance

The commercial aviation sector has achieved significant advancements in safety owing to robust Air Traffic Management technologies and rigorous regulatory measures. In contrast, General Aviation (GA) operations present unique safety challenges that demand focused attention. This study proposes an innovative decision support system tailored for GA pilots to augment their situational awareness. Our approach leverages on-board camera data in conjunction with semantic weather descriptors to construct an uncertainty-aware neural network model. The model provides predictions with quantified uncertainties while handling multiple labels and categories across diverse weather conditions. To validate the effectiveness of our framework, extensive experiments were conducted utilizing a flight simulator as a data collection platform. The results demonstrate that our model showcased significant improvements over the multiple baselines. We also found that a cost-sensitive learning approach can provide more conservative predictions while yielding performance improvements. Ultimately, our decision support framework aims to complement existing weather data sources, such as Next Generation Weather Radar (NEXRAD) data and Meteorological Aerodrome Reports (METAR) from airports, without imposing the burden of mounting expensive and bulky on-board weather radar systems.

Path Planning and Motion Control of Robot Dog Through Rough Terrain Based on Vision Navigation.

This article delineates the enhancement of an autonomous navigation and obstacle avoidance system for a quadruped robot dog. Part one of this paper presents the integration of a sophisticated multi-level dynamic control framework, utilizing Model Predictive Control (MPC) and Whole-Body Control (WBC) from MIT Cheetah. The system employs an Intel RealSense D435i depth camera for depth vision-based navigation, which enables high-fidelity 3D environmental mapping and real-time path planning. A significant innovation is the customization of the EGO-Planner to optimize trajectory planning in dynamically changing terrains, coupled with the implementation of a multi-body dynamics model that significantly improves the robot's stability and maneuverability across various surfaces. The experimental results show that the RGB-D system exhibits superior velocity stability and trajectory accuracy to the SLAM system, with a 20% reduction in the cumulative velocity error and a 10% improvement in path tracking precision. The experimental results also show that the RGB-D system achieves smoother navigation, requiring 15% fewer iterations for path planning, and a 30% faster success rate recovery in challenging environments. The successful application of these technologies in simulated urban disaster scenarios suggests promising future applications in emergency response and complex urban environments. Part two of this paper presents the development of a robust path planning algorithm for a robot dog on a rough terrain based on attached binocular vision navigation. We use a commercial-of-the-shelf (COTS) robot dog. An optical CCD binocular vision dynamic tracking system is used to provide environment information. Likewise, the pose and posture of the robot dog are obtained from the robot's own sensors, and a kinematics model is established. Then, a binocular vision tracking method is developed to determine the optimal path, provide a proposal (commands to actuators) of the position and posture of the bionic robot, and achieve stable motion on tough terrains. The terrain is assumed to be a gentle uneven terrain to begin with and subsequently proceeds to a more rough surface. This work consists of four steps: (1) pose and position data are acquired from the robot dog's own inertial sensors, (2) terrain and environment information is input from onboard cameras, (3) information is fused (integrated), and (4) path planning and motion control proposals are made. Ultimately, this work provides a robust framework for future developments in the vision-based navigation and control of quadruped robots, offering potential solutions for navigating complex and dynamic terrains.

Research on recognition of slippery road surface and collision warning system based on deep learning.

Aiming at the problems of slow detection speed, large prediction error and weak environmental adaptability of current vehicle collision warning system, this paper proposes a recognition method of slippery road surface and collision warning system based on deep learning. Firstly, this paper uses the on-board camera to monitor the environment and road conditions in front of the vehicle in real time, and a residual network model FS-ResNet50 is proposed, which integrated SE attention mechanism and multi-level feature information based on the traditional ResNet50 model. The FS-ResNet50 model is used to identify the slippery states of the current road, such as wet and snowy. Secondly, the yolov5 algorithm is used to detect the position of the vehicle in front, and a driving safety distance model with adaptive traffic environment characteristics is established based on different road conditions and driving conditions, and an early warning area that dynamically changed with the speed and the road slippery states is generated. Finally, according to the relationship between the warning area and the position of the vehicle, the possible collision is predicted and timely warned. Experimental results show that the method proposed in this paper improves the overall warning accuracy by 6.72% and reduces the warning false alarm rate for oncoming traffic on both sides by 16.67% compared with the traditional collision warning system. It can ensure safe driving, especially in bad weather conditions and has a high application value.

Autonomous UAV Chasing with Monocular Vision: A Learning-Based Approach

In recent years, unmanned aerial vehicles (UAVs) have shown significant potential across diverse applications, drawing attention from both academia and industry. In specific scenarios, UAVs are expected to achieve formation flying without relying on communication or external assistance. In this context, our work focuses on the classic leader-follower formation and presents a learning-based UAV chasing control method that enables a quadrotor UAV to autonomously chase a highly maneuverable fixed-wing UAV. The proposed method utilizes a neural network called Vision Follow Net (VFNet), which integrates monocular visual data with the UAV’s flight state information. Utilizing a multi-head self-attention mechanism, VFNet aggregates data over a time window to predict the waypoints for the chasing flight. The quadrotor’s yaw angle is controlled by calculating the line-of-sight (LOS) angle to the target, ensuring that the target remains within the onboard camera’s field of view during the flight. A simulation flight system is developed and used for neural network training and validation. Experimental results indicate that the quadrotor maintains stable chasing performance through various maneuvers of the fixed-wing UAV and can sustain formation over long durations. Our research explores the use of end-to-end neural networks for UAV formation flying, spanning from perception to control.

Application of Reinforcement Learning in Controlling Quadrotor UAV Flight Actions

Most literature has extensively discussed reinforcement learning (RL) for controlling rotorcraft drones during flight for traversal tasks. However, most studies lack adequate details regarding the design of reward and punishment mechanisms, and there is a limited exploration of the feasibility of applying reinforcement learning in actual flight control following simulation experiments. Consequently, this study focuses on the exploration of reward and punishment design and state input for RL. The simulation environment is constructed using AirSim and Unreal Engine, with onboard camera footage serving as the state input for reinforcement learning. The research investigates three RL algorithms suitable for discrete action training. The Deep Q Network (DQN), Advantage Actor–Critic (A2C), and Proximal Policy Optimization (PPO) were combined with three different reward and punishment design mechanisms for training and testing. The results indicate that employing the PPO algorithm along with a continuous return method as the reward mechanism allows for effective convergence during the training process, achieving a target traversal rate of 71% in the testing environment. Furthermore, this study proposes integrating the YOLOv7-tiny object detection (OD) system to assess the applicability of reinforcement learning in real-world settings. Unifying the state inputs of simulated and OD environments and replacing the original simulated image inputs with a maximum dual-target approach, the experimental simulation achieved a target traversal rate of 52% ultimately. In summary, this research formulates a set of logical frameworks for an RL reward and punishment design deployed with real-time Yolo’s OD implementation synergized as a useful aid for related RL studies.

Robust adversarial attacks detection for deep learning based relative pose estimation for space rendezvous

Research on developing deep learning techniques for autonomous spacecraft relative navigation challenges is continuously growing in recent years. Adopting those techniques offers enhanced performance. However, such approaches also introduce heightened apprehensions regarding the trustability and security of such deep learning methods through their susceptibility to adversarial attacks. In this work, we propose a novel approach for adversarial attack detection for deep neural network-based relative pose estimation schemes based on the explainability concept. We develop for an orbital rendezvous scenario an innovative relative pose estimation technique adopting our proposed Convolutional Neural Network (CNN), which takes an image from the chaser’s onboard camera and outputs accurately the target’s relative position and rotation. We perturb seamlessly the input images using adversarial attacks that are generated by the Fast Gradient Sign Method (FGSM). The adversarial attack detector is then built based on a Long Short Term Memory (LSTM) network which takes the explainability measure namely SHapley Value from the CNN-based pose estimator and flags the detection of adversarial attacks when acting. Simulation results show that the proposed adversarial attack detector achieves a detection accuracy of 99.21%. Both the deep relative pose estimator and adversarial attack detector are then tested on real data captured from our laboratory-designed setup. The experimental results from our laboratory-designed setup demonstrate that the proposed adversarial attack detector achieves an average detection accuracy of 96.29%.

Steering angle prediction in autonomous vehicles: a deep learning approach combining Vgg16 and Lstm

The field of autonomous driving has seen remarkable progress in recent years, with the prediction of steering angles based on varying road conditions emerging as a critical area of focus. While previous efforts have concentrated on lane detection, road object identification, and 3-D reconstruction, our research centers on a vision-based model that leverages deep networks to translate raw camera images into steering angles without the need for predefined feature learning. In our paper, we introduce an end-to-end model that employs deep transfer learning to predict steering angles from image sequences captured by onboard cameras. This model merges two deep learning architectures: a convolutional neural network (CNN) and a long short-term memory (LSTM) network. We utilize the VGG16 network, pre-trained on ImageNet and renowned for its performance, to extract spatial features from the images. Concurrently, the LSTM network processes the temporal information embedded within the image sequences. Our proposed model comprehensively processes spatial-temporal data and adeptly models the nonlinear relationship between the input images and the steering angles. We conducted an experimental study using a publicly available dataset to evaluate the model's effectiveness. The outcomes of our experimental analysis reveal that our model delivers highly efficient and accurate steering angle predictions, effectively emulating human driving patterns. Moreover, it surpasses current models in both performance and training efficiency, achieving a Mean Squared Error (MSE) score of 0.0728 as its loss function.

Analytical Second-Order Extended Kalman Filter for Satellite Relative Orbit Estimation

This study considers a relative orbit estimation problem wherein an observing spacecraft navigates with respect to a target space object at a large separation distance (several kilometers) using only the bearing angles obtained by a single onboard camera. Generally, the extended Kalman filter (EKF), which is based on linear relative motion equations such as the Clohessy–Wiltshire equation, is used for the relative navigation of satellites. The EKF linearizes the estimation error around the current estimate and applies the Kalman filter equations to this linearized system. However, it has been shown that nonlinearities of the orbit determination problem can make the linearization assumption insufficient to represent the actual uncertainty. Therefore, an analytical second-order extended Kalman filter (ASEKF) for relative orbit estimation is proposed in this study. The ASEKF, to sequentially estimate the relative states of satellites and their associated uncertainties, is formulated based on a second-order analytic relative-motion equation under J2-perturbtation, which can overcome the deficiencies of existing approaches that mainly focus on applications in two-body, near-circular, and linearized orbit dynamics. Numerical results show that the proposed method provides superior robustness and mean-square error performance compared to linear estimators under the conditions considered.

Rear passenger restraint in frontal NCAP tests compared to the right-front passenger

Objective This study compared kinematic and biomechanic responses of the 5th female Hybrid III in the right-rear and right-front passenger seats in frontal NCAP tests with 2015–16 MY vehicles. It focused on the lap-shoulder belt restraint of the rear passenger. Methods Eleven frontal NCAP tests were conducted by NHTSA at 56 km/h with a lap-shoulder belted 5th Hybrid III dummy in the right-rear and right-front seats. The right-front passenger had pretensioning and load limiting belts and advanced airbags with inflatable knee bolster. The rear passenger had seatbelts without pretensioning or load limiting. The kinematic and biomechanical responses of the 5th Hybrid III dummy in the rear and front were analyzed. The tests included onboard video cameras covering the rear and front passenger. Lap and shoulder belt loads were measured. The responses were compared for the two seating positions and restraint systems. Results Only one of 11 vehicles had favorable restraint of the right-rear passenger. All vehicles passed criteria for the right-front passenger. The right-rear passenger HIC15 was 888 ± 314 (779 IARV) significantly higher than 242 ± 72 in the right-front passenger (t = 6.36, p < 0.001). The right rear Nij was 1.16 ± 0.19 (1.00 IARV) significantly higher than 0.46 ± 0.11 in the right-front (t = 9.18, p < 0.001) and the rear chest deflection was 46.1 ± 9.8 mm (42 mm IARV) significantly higher than 20.8 ± 5.6 mm in the right-front (t = 7.12, p < 0.001). The shoulder belt load was 8,771 ± 2,088 N on the right-rear passenger significantly higher than 3,564 ± 911 N on the right-front (t = 7.28, p < 0.001). Three of the 11 crash tests involved submarining by the right-rear passenger at 76.0 ± 7.2 ms. Two tests with submarining involved the seatbelt buckle lifting by shoulder belt load pulling the lap belt onto the abdomen. One test involved compressing the front of the seat cushion with the H-point dropping, the pelvis rotating rearward and the lap belt sliding onto the abdomen. Conclusions Restraining loads on the rear passenger were significantly higher than on the right-front passenger in 2015–16 MY vehicle NCAP tests. The HIC15 was 3.7-times higher in the rear passenger than the front. The Nij was 2.5-times higher and chest deflection was 2.2-times higher. The dummy responses were significantly higher in the rear seat compared to the front seat because of greater restraining loads, submarining and a hardware failure in NCAP tests. The lack of pretensioning, load limiting and airbags involved higher injury risks in the rear passenger compared to the right-front passenger.

Study on Improvement of Docking Mechanism for Docking–Undocking Drones in the Air and Docking Control Using an Onboard Camera

In this study, we developed a docking mechanism for drones that enables docking and undocking in the air for cargo transportation. This capability allows the drone configuration to be adapted depending on the cargo size and shape. We introduced a novel docking mechanism that incorporates magnetic adsorption and a newly engineered locking mechanism. We developed a drone detection system capable of estimating the relative position of a target drone by utilizing an onboard camera to identify the infrared LEDs on the target drone via image processing. Moreover, flight experiments, including detection, docking, and undocking, were performed using a mock drone to demonstrate the effectiveness of the developed system.

Consecutive Image Acquisition without Anomalies.

An image is a visual representation that can be used to obtain information. A camera on a moving vector (e.g., on a rover, drone, quad, etc.) may acquire images along a controlled trajectory. The maximum visual information is captured during a fixed acquisition time when consecutive images do not overlap and have no space (or gap) between them. The images acquisition is said to be anomalous when two consecutive images overlap (overlap anomaly) or have a gap between them (gap anomaly). In this article, we report a new algorithm, named OVERGAP, that remove these two types of anomalies when consecutive images are obtained from an on-board camera on a moving vector. Anomaly detection and correction use here both the Dynamic Time Warping distance and Wasserstein distance. The proposed algorithm produces consecutive, anomaly-free images with the desired size that can conveniently be used in a machine learning process (mainly Deep Learning) to create a prediction model for a feature of interest.

Online extrinsic parameters calibration of on-board stereo cameras based on certifiable optimization

The extrinsic parameters of on-board stereo cameras can be slightly altered due to temperature fluctuations, vibrations, and accidental impacts during automobile driving, leading to significant performance loss in dense stereo matching. In this paper, we propose an online calibration method based on certifiable optimization to address this issue. Initially, sparse feature points are collected based on plane distribution and disparity. A robust optimization model is then developed to minimize the epipolar error, utilizing iterative local optimization to eliminate outliers and determine the 5DOF extrinsic parameters. Subsequently, a relaxation problem is constructed using inliers, and global optimization is performed to certify that the locally optimal results are indeed globally optimal. Comparative experimental results demonstrate that the proposed method offers high accuracy and reliability. Additionally, the quality of the disparity map generated by our calibration method is comparable to that achieved through offline calibration.

Autonomous Multirotor UAV Search and Landing on Safe Spots Based on Combined Semantic and Depth Information From an Onboard Camera and LiDAR

Autonomous Multirotor UAV Search and Landing on Safe Spots Based on Combined Semantic and Depth Information From an Onboard Camera and LiDAR

Convolutional-Neural-Network-Based Autonomous Navigation of Hera Mission Around Didymos

The European Space Agency (ESA)’s Hera mission requires autonomous visual-based navigation in order to safely orbit around the target binary asteroid system Didymos and its moon Dimorphos in 2027. Nevertheless, the performance of optical-based navigation systems depends on the intrinsic properties of the image, such as high Sun phase angles, the presence of other bodies, and, especially, the irregular shape of the target. Therefore, to improve the navigation performance, thermal and/or range measurements from additional onboard instruments are usually needed to complement optical measurements. However, this work addresses these challenges by developing a fully visual-based autonomous navigation system using a convolutional-neural-network (CNN)-based image processing (IP) algorithm and applying it to the detailed characterization phase of the proximity operation of the mission. The images taken by the onboard camera are processed by the CNN-based IP algorithm that estimates the position of the geometrical centers of Didymos and Dimorphos, the range from Didymos, and the associated covariances. The results show that the developed algorithm can be used for a fully visual-based navigation for the position of the Hera spacecraft around the target with good robustness and accuracy.

Object Detection and Classification Framework for Analysis of Video Data Acquired from Indian Roads

Object detection and classification in autonomous vehicles are crucial for ensuring safe and efficient navigation through complex environments. This paper addresses the need for robust detection and classification algorithms tailored specifically for Indian roads, which present unique challenges such as diverse traffic patterns, erratic driving behaviors, and varied weather conditions. Despite significant progress in object detection and classification for autonomous vehicles, existing methods often struggle to generalize effectively to the conditions encountered on Indian roads. This paper proposes a novel approach utilizing the YOLOv8 deep learning model, designed to be lightweight, scalable, and efficient for real-time implementation using onboard cameras. Experimental evaluations were conducted using real-life scenarios encompassing diverse weather and traffic conditions. Videos captured in various environments were utilized to assess the model’s performance, with particular emphasis on its accuracy and precision across 35 distinct object classes. The experiments demonstrate a precision of 0.65 for the detection of multiple classes, indicating the model’s efficacy in handling a wide range of objects. Moreover, real-time testing revealed an average accuracy exceeding 70% across all scenarios, with a peak accuracy of 95% achieved in optimal conditions. The parameters considered in the evaluation process encompassed not only traditional metrics but also factors pertinent to Indian road conditions, such as low lighting, occlusions, and unpredictable traffic patterns. The proposed method exhibits superiority over existing approaches by offering a balanced trade-off between model complexity and performance. By leveraging the YOLOv8 architecture, this solution achieved high accuracy while minimizing computational resources, making it well suited for deployment in autonomous vehicles operating on Indian roads.

Hybrid Visual Odometry Algorithm Using a Downward-Facing Monocular Camera

The increasing interest in developing robots capable of navigating autonomously has led to the necessity of developing robust methods that enable these robots to operate in challenging and dynamic environments. Visual odometry (VO) has emerged in this context as a key technique, offering the possibility of estimating the position of a robot using sequences of onboard cameras. In this paper, a VO algorithm is proposed that achieves sub-pixel precision by combining optical flow and direct methods. This approach uses only a downward-facing, monocular camera, eliminating the need for additional sensors. The experimental results demonstrate the robustness of the developed method across various surfaces, achieving minimal drift errors in calculation.

Heliostat geometrical characterization by UAV-assisted, close-range photogrammetry

An innovative approach for the characterization of the mirror geometry of Concentrating Solar Thermal (CST) plants is presented in this article. The approach takes advantage of the current advances in the commercial unmanned aerial vehicle (UAV) technology to perform airborne, close-range photogrammetry for accurate characterization of the mirror’s geometry. To improve spatial resolution and, thus, reconstruction performance, the approach involves a projector for projecting a feature-rich image on the surface of the mirror to be characterized. A UAV flying on a predetermined flight path is used to capture overlapping snapshots of the projected image using its on-board high definition digital camera. By the use of photogrammetry tools, the snapshots are stitched together and processed accordingly to reconstruct the surface of the reflector. Although the approach was tested and validated experimentally by characterizing the mirror surface of a single-facet heliostat, the developed method can also be applied for the characterization of any type of solar concentrating mirror geometry.

Defog YOLO for road object detection in foggy weather

Abstract Object detection research predominantly focuses on clear weather conditions, often overlooking the challenges posed by foggy weather. Fog impairs the vision of onboard cameras, creating significant obstacles for autonomous vehicles. To tackle these issues, we present the Defog YOLO algorithm, specifically designed for road object detection in foggy conditions. Our approach integrates an enhanced U-Net framework for visual defogging, where the encoder leverages super-resolution back projection to combine multi-layer features. The decoder employs a back projection feedback mechanism to improve image restoration. Additionally, we augment the Feature Pyramid Network with a noise-aware attention mechanism, allowing the network to emphasize critical channel and spatial information while mitigating noise. Given the scarcity of labeled foggy images, we introduce a fog addition module to generate a more diverse training dataset. We validate our method using a synthesized FOG-TRAINVAL dataset, derived from the VOC dataset, demonstrating its robustness in foggy scenarios. Experimental results show that our proposed method achieves an mAP score of 60% on the Real-world Task-driven Testing Set foggy weather test set, with a precision of 86.7% and a recall of 54.2%. These findings underscore the effectiveness and improved generalizability of our approach for object detection in adverse weather conditions.

CycleInSight: An enhanced YOLO approach for vulnerable cyclist detection in urban environments

As urbanization continues to reshape transportation, the safety of cyclists in complex traffic environments has become a pressing concern. In response to this challenge, our research introduces a CycleInSight framework, which harnesses advanced deep learning and computer vision techniques to enable precise and efficient cyclist detection in diverse urban settings. Utilizing you only look once version 8 (YOLOv8) object detection algorithm, the proposed model aims to detect and localize vulnerable cyclists near vehicles equipped with onboard cameras. Our research presents comprehensive experimental results demonstrating its effectiveness in identifying vulnerable cyclists amidst dynamic and challenging traffic conditions. With an impressive average precision of 90.91%, our approach outperforms existing models while maintaining efficient inference speeds. By effectively identifying and tracking cyclists, this framework holds significant potential to enhance urban traffic safety, inform data-driven infrastructure planning, and support the development of advanced driver assistance systems and autonomous vehicles.

A robust method for multi object tracking in autonomous ship navigation systems

A novel Maritime Multi-Object Tracking method is proposed, combining a deep learning-based object detector with target association algorithms to achieve robust sea-surface multi-object tracking. Specifically, the Multi-Object Tracking employs You Only Look Once version 7 detector for object detection. In the data association part, a module for onboard camera motion compensation is developed, a maritime dynamic spatial information-based intersection-over-union is presented as a similarity metric, and a progressive refinement cascade matching strategy is designed to enhance the tracker's sea-surface multi-object tracking capabilities. The Jari Maritime Tracking Dataset is utilised to validate the effectiveness performance of the proposed method. Experimental results demonstrate that compared to the earlier process, the proposed method exhibits a significant enhancement in multiple object tracking accuracy, with an increase of 27.8% and an achieved score of 81.3. In particular, it reduces the number of identifications switching and missed targets, achieving holistically preferable performance. Meanwhile, the speed of the proposed Multi-Object Tracking fulfils the engineering application requirements for an autonomous ship navigation system.