Extreme Lighting Conditions Research Articles

Comparative Metabolomic Analysis Reveals the Impact of the Photoperiod on the Hepatopancreas of Chinese Grass Shrimp (Palaemonetes sinensis)

The photoperiod is a key environmental factor that in crustaceans influences development, feeding, and metabolism. In this study, liquid chromatography-tandem mass spectrometry was used to examine metabolic changes in Palaemonetes sinensis under different photoperiods. Our results showed that key metabolic pathways, such as linoleic acid metabolism, axon regeneration, pyrimidine metabolism, and cortisol synthesis, were significantly altered under both constant light (24L:0D) and constant darkness (0L:24D) compared with natural light conditions. The photoperiod notably affected the digestive and metabolic functions of P. sinensis. Most metabolic pathways were downregulated under full darkness and full light conditions, suggesting that inhibition of metabolism is a potential adaptive response. Furthermore, enzyme assays revealed significant variations in trypsin, lipase, and amylase activity across different photoperiods, highlighting the profound impact of light conditions on digestive functions. These findings suggest that extreme light conditions may negatively impact the metabolic and digestive functions of P. sinensis. This study provides new insights into the adaptive mechanisms of P. sinensis in response to photoperiod changes and offers valuable information for optimizing aquaculture practices to enhance the health and growth performance of this crustacean.

Hierarchical Event-RGB Interaction Network for single-eye expression recognition

The Single-eye Expression Recognition task stands as a crucial vision task, aimed at decoding human emotional states through careful examination of the eye region. Nevertheless, traditional cameras face challenges in detecting and capturing relevant biological information, especially under demanding lighting conditions such as dim environments, high exposure scenarios, or when other radiation sources are present. In this regard, we use a new type of sensor data that can resist extreme lighting conditions, namely event camera data, to improve the performance of single-eye expression recognition. To this end, we propose a novel Hierarchical Event-RGB Interaction Network (HI-Net), to fully integrate RGB and event data to overcome the extreme lighting challenges faced by the single-eye expression recognition task. The HI-Net contains two novel designs: Event-RGB Semantic Interaction Mechanism (ER-SIM) and Hierarchical Semantics Modeling (HSM) Scheme. The former aims to achieve interaction between Event and RGB modality semantics, while the latter aims to obtain high-quality modality semantic representations. In the ER-SIM, we employ an effective cross-attention mechanism to facilitate information fusion, to adaptively integrate and complement multi-scale Event and RGB semantics to cope with extreme lighting conditions. In HSM Scheme, we first explore multi-scale contextual semantics for the event modality and the RGB modality respectively. Then, we perform a semantics interaction strategy for these multi-scale contextual semantics, to enhance each modality's semantic representation. Extensive experiments demonstrate that our HI-Net significantly outperforms many state-of-the-art methods on the single-eye expression recognition task, especially under degraded lighting conditions.

"Seeing" ENF From Neuromorphic Events: Modeling and Robust Estimation.

Most artificial lights exhibit subtle fluctuations in intensity and frequency in response to the influence of the grid's alternating current, providing the potential to estimate the Electric Network Frequency (ENF) from conventional frame-based videos. Nevertheless, the performance of Video-based ENF (V-ENF) estimation largely relies on the imaging quality and thus may suffer from significant interference caused by non-ideal sampling, scene diversity, motion interference, and extreme lighting conditions. In this paper, we show that the ENF can be extracted without the above limitations from a new modality provided by the so-called event camera, a neuromorphic sensor that encodes the light intensity variations and asynchronously emits events with extremely high temporal resolution and high dynamic range. Specifically, we formulate and validate the physical mechanism for the ENF captured in events and then propose a simple yet robust Event-based ENF (E-ENF) estimation method through mode filtering and harmonic enhancement. To validate the effectiveness, we build the first Event-Video ENF Dataset (EV-ENFD) and its extension EV-ENFD+ with diverse scenarios, including static, dynamic, and extreme lighting scenes. Comprehensive experiments have been conducted on our proposed datasets, showcasing that our proposed E-ENF significantly outperforms the V-ENF in extracting accurate ENF traces, especially in challenging environments.

IMPLEMENTASI ALGORITMA KNN UNTUK DETEKSI GENDER DAN USIA BERBASIS PENGENALAN WAJAH

In the modern era, face recognition is one of the important technologies used in various security and data analysis applications. This paper aims to use Python to implement the K-Nearest Neighbors (KNN) algorithm to detect gender and age through face recognition. The main problem faced is the accuracy in classifying gender and estimating age from various facial images with lighting, pose, and expression. This study uses various steps to collect facial data, use image processing techniques to extract features, and use the KNN algorithm for classification. The results show that applying KNN can detect gender and age from faces with sufficient accuracy, although there are some problems with extreme lighting and pose conditions. These findings indicate that the KNN algorithm can be used in facial recognition applications to detect gender and age. By fixing parameters and improving data quality, further improvements can be achieved.

Dimma: Semi-Supervised Low-Light Image Enhancement with Adaptive Dimming.

Enhancing low-light images with natural colors poses a challenge due to camera processing variations and limited access to ground-truth lighting conditions. To address this, we propose Dimma, a semi-supervised approach that aligns with any camera using a small set of image pairs captured under extreme lighting conditions. Our method employs a convolutional mixture density network to replicate camera-specific noise present in dark images. We enhance results further by introducing a conditional UNet architecture based on user-provided lightness values. Trained on just a few real image pairs, Dimma achieves competitive results compared to fully supervised state-of-the-art methods trained on large datasets.

YOLO-Based Vehicle Detection: Literature Review

This research aims to evaluate the implementation of the You Only Look Once (YOLO) algorithm and its variants in the context of vehicle detection in traffic management systems. The importance of implementing intelligent transportation systems (ITS) in increasing transportation efficiency and reducing traffic problems such as congestion and accidents. The methodology used involves a critical review of current literature utilizing the YOLO algorithm for vehicle detection, with a focus on improving the accuracy of detection models. The research results show that the YOLO algorithm and its variants, such as YOLOv4 and YOLOv8, show a significant increase in vehicle detection accuracy reaching 90% in various environmental conditions. However, weaknesses in detecting small objects and in extreme lighting conditions still need further attention. This study also reviews several improvement approaches proposed in the literature, including the use of image augmentation techniques and the integration of deep learning models to improve the performance of the YOLO algorithm. The implementation of the YOLO algorithm in vehicle detection in intelligent transportation systems has great potential in increasing the efficiency and accuracy of traffic monitoring. This research provides recommendations for further development so that the YOLO algorithm can be better adapted to various environmental conditions and different types of data.

Multi-model fusion of DSFD and Zero-DCE for low-light face detection

Due to the wide application of face detection, face recognition task under extreme lighting conditions inevitably becomes one of the challenges. Under low-light conditions, the face detection task becomes more complex and difficult due to the presence of more noise and lower visibility in the image. For this extreme environment, this paper proposes a multi-model-assisted solution that combines image enhancement and face detection models to handle this task. The DarkFace dataset was used in this study's validation of a number of distinct face identification models and picture enhancement techniques. The experimental findings demonstrate that, on this dataset, the DSFD (Dual Shot Face Detector) and Zero-DCE (Zero-Reference Deep Curve Estimation) methods perform noticeably better than the other techniques, achieving a recall of 24.61%. Therefore, these two algorithms are selected as the core processing flow in this paper to improve the facial detection system's low-light performance. This combination offers a workable answer to the problem of face detection in low light by effectively increasing the visibility of the picture and raising the accuracy of face detection.

WH-DETR: An Efficient Network Architecture for Wheat Spike Detection in Complex Backgrounds

Wheat spike detection is crucial for estimating wheat yields and has a significant impact on the modernization of wheat cultivation and the advancement of precision agriculture. This study explores the application of the DETR (Detection Transformer) architecture in wheat spike detection, introducing a new perspective to this task. We propose a high-precision end-to-end network named WH-DETR, which is based on an enhanced RT-DETR architecture. Initially, we employ data augmentation techniques such as image rotation, scaling, and random occlusion on the GWHD2021 dataset to improve the model’s generalization across various scenarios. A lightweight feature pyramid, GS-BiFPN, is implemented in the network’s neck section to effectively extract the multi-scale features of wheat spikes in complex environments, such as those with occlusions, overlaps, and extreme lighting conditions. Additionally, the introduction of GSConv enhances the network precision while reducing the computational costs, thereby controlling the detection speed. Furthermore, the EIoU metric is integrated into the loss function, refined to better focus on partially occluded or overlapping spikes. The testing results on the dataset demonstrate that this method achieves an Average Precision (AP) of 95.7%, surpassing current state-of-the-art object detection methods in both precision and speed. These findings confirm that our approach more closely meets the practical requirements for wheat spike detection compared to existing methods.

Cross-modal learning for optical flow estimation with events

Benefiting from the low latency and high dynamic range, event cameras have recently been adopted for Optical Flow (OF) prediction under harsh environments with high-speed motion or extreme lighting conditions. However, the emitted events only respond to brightness changes, mostly at edges with high-intensity contrast, leading to the ill-posedness of OF prediction in non-edge or low-contrast regions. To address this problem, we propose a network to learn OF by exploring the complements between the frame and the event domains. In particular, we first extract cross-modal features from frames and events via a fusion-based encoder and then apply a self-attention module of cross-dimension interaction to suppress disturbances in noisy events. Furthermore, a long-term memory recurrent module is built for local and global temporal information combination and cross-modal feature enhancement. Finally, a CNN decoder is designed to upscale cross-modal features and provide pyramidal results for multi-scale loss computation. Extensive experiments demonstrate that the proposed method can achieve leading performance compared with the state-of-the-art OF models.

Ultra Fast Deep Lane Detection With Hybrid Anchor Driven Ordinal Classification.

Modern methods mainly regard lane detection as a problem of pixel-wise segmentation, which is struggling to address the problems of efficiency and challenging scenarios like severe occlusions and extreme lighting conditions. Inspired by human perception, the recognition of lanes under severe occlusions and extreme lighting conditions is mainly based on contextual and global information. Motivated by this observation, we propose a novel, simple, yet effective formulation aiming at ultra fast speed and the problem of challenging scenarios. Specifically, we treat the process of lane detection as an anchor-driven ordinal classification problem using global features. First, we represent lanes with sparse coordinates on a series of hybrid (row and column) anchors. With the help of the anchor-driven representation, we then reformulate the lane detection task as an ordinal classification problem to get the coordinates of lanes. Our method could significantly reduce the computational cost with the anchor-driven representation. Using the large receptive field property of the ordinal classification formulation, we could also handle challenging scenarios. Extensive experiments on four lane detection datasets show that our method could achieve state-of-the-art performance in terms of both speed and accuracy. A lightweight version could even achieve 300+ frames per second(FPS). Our code is at https://github.com/cfzd/Ultra-Fast-Lane-Detection-v2.

An Autonomous Navigation Method for Planetary Rover Based on Multi-modal Fusion and Multi-factor Graph Optimization

Planetary surface is a complex unstructured environment with strong light, strong shadows, weak textures and numerous obstacles. There is no GPS signal support and satellite networking support on the planetary surface. The limited computer and sensor capabilities of planetary rover have a significant gap from the ground autonomous driving configuration, which presents a great challenge to the autonomous navigation methods of planetary rover. Extreme lighting conditions will increase the camera mismatching rate, and poor texture will increase the lidar matching error, which will reduce the estimation accuracy. Moreover, camera and lidar produce great motion distortion in the rugged terrain and bumpy environment, which leads to a large increase in the cumulative error of odometer. Camera and lidar are not suitable as the main sensor of SLAM algorithm in unstructured environment because they are sensitive to the change of environment. Therefore, an autonomous navigation method for planetary rover based on multi-modal fusion and multi-factor graph optimization is presented in this paper. The IMU (Inertial Measurement Unit) odometer node is added to the lidar odometer and visual odometer thread, and the IMU is taken as the central node to build a multi-factor graph optimization model. A multi-factor graph optimization model and a strong adaptive constraint strategy with reasonable weights are constructed. The pose estimation of other sensors is used to constrain IMU bias, meanwhile historical rut tracking, obstacle contour matching and skyline feature matching are introduced. Finally, motion prediction is achieved in the IMU odometer node. Simulation results and field tests show that this method can effectively cope with the unstructured environment on planetary surface, which achieves adaptive and robust autonomous navigation of planetary rover.

Laryngeal surface reconstructions from monocular endoscopic videos: a structure from motion pipeline for periodic deformations.

Surface reconstructions from laryngoscopic videos have the potential to assist clinicians in diagnosing, quantifying, and monitoring airway diseases using minimally invasive techniques. However, tissue movements and deformations make these reconstructions challenging using conventional pipelines. To facilitate such reconstructions, we developed video frame pre-filtering and featureless dense matching steps to enhance the Alicevision Meshroom SfM pipeline. Time and the anterior glottic angle were used to approximate the rigid state of the airway and to collect frames with different camera poses. Featureless dense matches were tracked with a correspondence transformer across subsets of images to extract matched points that could be used to estimate the point cloud and reconstructed surface. The proposed pipeline was tested on a simulated dataset under various conditions like illumination and resolution as well as real laryngoscopic videos. Our pipeline was able to reconstruct the laryngeal region based on 4, 8, and 16 images obtained from simulated and real patient exams. The pipeline was robust to sparse inputs, blur, and extreme lighting conditions, unlike the Meshroom pipeline which failed to produce a point cloud for 6 of 15 simulated datasets. The pre-filtering and featureless dense matching modules specialize the conventional SfM pipeline to handle the challenging laryngoscopic examinations, directly from patient videos. These 3D visualizations have the potential to improve spatial understanding of airway conditions.

Experimental study on visual ergonomics of an aircraft cockpit considering an extremely wide range of illuminance conditions

Vision is the main way for pilots to obtain information, and good visual ergonomics are an important support for ensuring aircraft flight safety. The range of illumination changes in the light environment of the aircraft cockpit is very wide, and research on the visual ergonomics of the cockpit needs to consider various extreme lighting conditions. This study conducted visual ergonomics experiments on 15 participants in a full-scale simulated cockpit, examining the accuracy, reaction time, and subjective evaluation of visual tasks under 8 typical environmental lighting intensity levels. The experimental results show that, except for head-up display, the accuracy of visual target interpretation tasks performed by other display devices under different brightness conditions remains at a high level. And as the brightness of the display device increases, the accuracy of interpretation gradually increases, and the reaction time gradually decreases. In terms of subjective evaluation, there is a significant correlation between fuzziness, fatigue, clarity of image symbols, resolution between symbols, comfort of the image, and overall satisfaction with the image, but the correlation with environmental illumination level is relatively low. The experimental results can provide a certain theoretical basis for the design of cockpit lighting environment.

On-ground validation of orbital GNC: Visual navigation assessment in robotic testbed facility

AbstractCubeSats have become versatile platforms for various space missions (e.g., on-orbit servicing and debris removal) owing to their low cost and flexibility. Many space tasks involve proximity operations that require precise guidance, navigation, and control (GNC) algorithms. Vision-based navigation is attracting interest for such operations. However, extreme lighting conditions in space challenge optical techniques. The on-ground validation of such navigation systems for orbital GNC becomes crucial to ensure their reliability during space operations. These systems undergo rigorous testing within their anticipated operational parameters, including the exploration of potential edge cases. The ability of GNC algorithms to function effectively under extreme space conditions that exceed anticipated scenarios is crucial, particularly in space missions where the scope of errors is negligible. This paper presents the ground validation of a GNC algorithm designed for autonomous satellite rendezvous by leveraging hardware-in-the-loop experiments. This study focuses on two key areas. First, the rationale underlying the augmentation of the robot workspace (six-degree-of-freedom UR10e robot + linear rail) is investigated to emulate relatively longer trajectories with complete position and orientation states. Second, the control algorithm is assessed in response to uncertain pose observations from a vision-based navigation system. The results indicate increased control costs with uncertain navigation and exemplify the importance of on-ground testing for system validation before launch, particularly in extreme cases that are typically difficult to assess using software-based testing.

ST-LaneNet: Lane Line Detection Method Based on Swin Transformer and LaneNet

The advancement of autonomous driving heavily relies on the ability to accurate lane lines detection. As deep learning and computer vision technologies evolve, a variety of deep learning-based methods for lane line detection have been proposed by researchers in the field. However, owing to the simple appearance of lane lines and the lack of distinctive features, it is easy for other objects with similar local appearances to interfere with the process of detecting lane lines. The precision of lane line detection is limited by the unpredictable quantity and diversity of lane lines. To address the aforementioned challenges, we propose a novel deep learning approach for lane line detection. This method leverages the Swin Transformer in conjunction with LaneNet (called ST-LaneNet). The experience results showed that the true positive detection rate can reach 97.53% for easy lanes and 96.83% for difficult lanes (such as scenes with severe occlusion and extreme lighting conditions), which can better accomplish the objective of detecting lane lines. In 1000 detection samples, the average detection accuracy can reach 97.83%, the average inference time per image can reach 17.8 ms, and the average number of frames per second can reach 64.8 Hz. The programming scripts and associated models for this project can be accessed openly at the following GitHub repository: https://github.com/Duane711/Lane-line-detection-ST-LaneNet.

Continuous-time Object Segmentation using High Temporal Resolution Event Camera.

Event cameras are novel bio-inspired sensors, where individual pixels operate independently and asynchronously, generating intensity changes as events. Leveraging the microsecond resolution (no motion blur) and high dynamic range (compatible with extreme light conditions) of events, there is considerable promise in directly segmenting objects from sparse and asynchronous event streams in various applications. However, different from the rich cues in video object segmentation, it is challenging to segment complete objects from the sparse event stream. In this paper, we present the first framework for continuous-time object segmentation from event stream. Given the object mask at the initial time, our task aims to segment the complete object at any subsequent time in event streams. Specifically, our framework consists of a Recurrent Temporal Embedding Extraction (RTEE) module based on a novel ResLSTM, a Cross-time Spatiotemporal Feature Modeling (CSFM) module which is a transformer architecture with long-term and short-term matching modules, and a segmentation head. The historical events and masks (reference sets) are recurrently fed into our framework along with current-time events. The temporal embedding is updated as new events are input, enabling our framework to continuously process the event stream. To train and test our model, we construct both real-world and simulated event-based object segmentation datasets, each comprising event streams, APS images, and object annotations. Extensive experiments on our datasets demonstrate the effectiveness of the proposed recurrent architecture. Our code and dataset are available at https://sites.google.com/view/ecos-net/.

Hundreds-petawatt laser pulses shaping and heavy ion acceleration based on conical plasma channels

In this paper, the effects of conical plasma channels on the laser pulses shaping and the heavy ion acceleration under the extreme light field conditions of hundreds-petawatt are investigated using a particle simulation method. The law of influence of the conical plasma channel on the spatio-temporal waveform and intensity of the incident laser is analyzed, when the QED effect is taken into account. The reason for the shaping laser-enhanced heavy ion acceleration is given, and the role of the QED effect in the acceleration process is explained.<br>It is found that, due to the non-linear interference and focusing effects, the conical plasma channel can shape the spatio-temporal waveform of the laser pulse and enhance the laser intensity. A tightly focused (beam waist radius < 1 μm) and ultra-high intensity (enhanced 6 times) shaping laser is obtained for a linearly polarized laser with an intensity of 5.46 × 10<sup>22</sup> W/cm<sup>2</sup> and a waist radius of 10 μm at an incident angle of <i>θ</i> = 10°. In the simulation, the conical plasma channel is fully ionized high-Z gold plasma with the electron densities up to <i>n<sub>e</sub></i>=2626.5<i>n<sub>c</sub></i>. Therefore most of the laser energy in the channel is reflected by the channel wall, and the QED effect has less impact on laser focusing and shaping. Using this laser to accelerate an ultra-thin flat target placed at the end of the channel. It is found that, the radiation reaction force can effectively suppress the transverse expansion of the ultra-thin flat target caused by the electron heating and the transverse non-uniform of the laser intensity. The transparency time of the ultra-thin flat target is prolonged, which will allow the gold ions to be fully accelerated. Ultimately, gold ions with a cutoff energy of up to ~ 240 GeV can be obtained. The results are expected to provide theoretical references and technical support for the design of future hundreds-petawatt laser heavy ion acceleration experiments and their application research of high-quality ion source, such as nucleus-nucleus collisions.

Involvement of Dynamic Adjustment of ABA, Proline and Sugar Levels in Rhizomes in Effective Acclimation of Solidago gigantea to Contrasting Weather and Soil Conditions in the Country of Invasion.

Giant goldenrod (Solidago gigantea Aiton) is one of the most invasive plant species occurring in Europe. Since little is known about the molecular mechanisms contributing to its invasiveness, we examined the natural dynamics of the content of rhizome compounds, which can be crucial for plant resistance and adaptation to environmental stress. We focused on rhizomes because they are the main vector of giant goldenrod dispersion in invaded lands. Water-soluble sugars, proline, and abscisic acid (ABA) were quantified in rhizomes, as well as ABA in the rhizosphere from three different but geographically close natural locations in Poland (50°04'11.3″ N, 19°50'40.2″ E) under extreme light, thermal, and soil conditions, in early spring, late summer, and late autumn. The genetic diversity of plants between locations was checked using the random amplified polymorphic DNA (RAPD) markers. Sugar and proline content was assayed spectrophotometrically, and abscisic acid (ABA) with the ELISA immunomethod. It can be assumed that the accumulation of sugars in giant goldenrod rhizomes facilitated the process of plant adaptation to adverse environmental conditions (high temperature and/or water scarcity) caused by extreme weather in summer and autumn. The same was true for high levels of proline and ABA in summer. On the other hand, the lowering of proline and ABA in autumn did not confirm the previous assumptions about their synthesis in rhizomes during the acquisition of frost resistance by giant goldenrod. However, in the location with intensive sunlight and most extreme soil conditions, a constant amount of ABA in rhizomes was noticed as well as its exudation into the rhizosphere. This research indicates that soluble sugars, proline, and ABA alterations in rhizomes can participate in the mechanism of acclimation of S. gigantea to specific soil and meteorological conditions in the country of invasion irrespective of plant genetic variation.

Bionic visual navigation model for enhanced template matching and loop closing in challenging lighting environments

In recent years, various computational models have been proposed in neuroscience, but only a few have been utilized in robot visual navigation through engineering approaches. Besides, these engineering methods lack environmental adaptability, especially in the aspect of visual place recognition in weak or uncontrollable light fluctuations. To address this issue and enhance the performance of visual template matching and map loop closure detection in challenging lighting environments, this paper proposed a bionic visual navigation model that combines two neural network architectures, the Pulse Coupled Neural Network (PCNN) and Continuous Attractor Neural Network (CANN). In our navigation model, the visual features of the environments are encoded as temporal information by the spiking model and input into the local view cells for visual template matching. Additionally, we utilize the pose cells network, which incorporates the similarity between current and previous templates, along with odometry data, to encode spatial information and store it as an experience map. To validate the effectiveness of the proposed model, we conducted evaluations on datasets collected within our library, campus, and an open-source office dataset. The experimental results reveal that our algorithm increases the F1-score of template matching by approximately 10.5%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$ \\% $$\\end{document}, 35.8%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$ \\% $$\\end{document}, 61.7%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$ \\% $$\\end{document}, and 1.9%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$ \\% $$\\end{document} in each dataset, compared to the conventional RatSLAM method. Furthermore, our algorithm generates a more accurate map that closely correlates with the real-world trajectory without compromising on computation time. The results suggest that our bionic visual navigation model is reliable for both standard and extreme lighting conditions.

SRVIO: Super Robust Visual Inertial Odometry for Dynamic Environments and Challenging Loop-Closure Conditions

There has been extensive research on visual localization and odometry for autonomous robots and virtual reality during the past decades. Traditionally, this problem has been solved with the help of expensive sensors, such as light detection and ranging (LiDAR). Nowadays, the focus of the leading research in this field is on robust localization using more economic sensors, such as cameras and inertial measurement units. Consequently, geometric visual localization methods have become more accurate over time. However, these methods still suffer from significant loss and divergence in challenging environments, such as a room full of moving people. Scientists started using deep neural networks (DNNs) to mitigate this problem. The main idea behind using DNNs is to better understand challenging aspects of the data and overcome complex conditions such as the movement of a dynamic object in front of the camera that covers the full view of the camera, extreme lighting conditions, and the high speed of the camera. Prior end-to-end DNN methods did overcome some of these challenges. However, no general and robust framework is available to overcome all challenges together. In this article, we have combined geometric and DNN-based methods to have the generality and speed of geometric SLAM frameworks and overcome most of these challenging conditions with the help of DNNs and deliver the most robust framework so far. To do so, we have designed a framework based on VINS-Mono and shown that it can achieve state-of-the-art results on TUM-Dynamic, TUM-VI, ADVIO, and EuRoC datasets compared to geometric and end-to-end DNN-based simultaneous localization and mappings. Our proposed framework can also achieve outstanding results on extreme simulated cases resembling the aforementioned challenges.