Adverse Lighting Conditions Research Articles

Siam-AUnet: An end-to-end infrared and visible image fusion network based on gray histogram

Infrared and visible image fusion aim to leverage the respective advantages of both modalities to extract richer information from a scene. In environments with ample lighting, it is imperative to preserve the characteristics of visible images and salvage details lost due to overexposure. Conversely, in dark environments, a preference is typically inclined towards the utilization of infrared imagery. However, current methodologies are unable to adaptively modulate their tendency towards either modality, thereby resulting in suboptimal fusion quality. A gray histogram visually represents the dynamic range of image distribution, revealing the contrast and degree of brightness variations. Drawing from this insight, in this paper, we propose a novel image fusion method called Siam-AUnet for the first time. This methodology transforms the distribution of luminance histograms extracted from visible images into weights during the fusion of the two modalities. We adopt a Siamese network architecture and a multi-scale attention mechanism to focus more effectively on key features while reducing the number of parameters. Extensive experiments on different datasets and comparisons with other methods have demonstrated the superiority of this fusion approach. By utilizing the characteristics of the histogram, Siam-AUnet can adaptively tend towards the modality with clearer texture details. Qualitative and quantitative experimental results on the MSRS test dataset demonstrate that Siam-AUnet achieves optimal performance in terms of EN, SSIM, and MS_SSIM, while exhibiting suboptimal performance in metrics such as SCD, VIF, and Qqbf. In well-lit conditions, it can preserve the detailed features of visible images and the salient features of infrared images; while under adverse lighting conditions, it emphasizes the infrared modality features more strongly. Our code will be publicly available at https://github.com/xkangKK/Siam-AUnet.

Aleth-NeRF: Illumination Adaptive NeRF with Concealing Field Assumption

The standard Neural Radiance Fields (NeRF) paradigm employs a viewer-centered methodology, entangling the aspects of illumination and material reflectance into emission solely from 3D points. This simplified rendering approach presents challenges in accurately modeling images captured under adverse lighting conditions, such as low light or over-exposure. Motivated by the ancient Greek emission theory that posits visual perception as a result of rays emanating from the eyes, we slightly refine the conventional NeRF framework to train NeRF under challenging light conditions and generate normal-light condition novel views unsupervisedly. We introduce the concept of a ``Concealing Field," which assigns transmittance values to the surrounding air to account for illumination effects. In dark scenarios, we assume that object emissions maintain a standard lighting level but are attenuated as they traverse the air during the rendering process. Concealing Field thus compel NeRF to learn reasonable density and colour estimations for objects even in dimly lit situations. Similarly, the Concealing Field can mitigate over-exposed emissions during rendering stage. Furthermore, we present a comprehensive multi-view dataset captured under challenging illumination conditions for evaluation. Our code and proposed dataset are available at https://github.com/cuiziteng/Aleth-NeRF.

A Dual-Branch Autoencoder Network for Underwater Low-Light Polarized Image Enhancement

Underwater detection faces uncomfortable illumination conditions, and traditional optical images sensitive to intensity often cannot work well in these conditions. Polarization imaging is a good solution for underwater detection under adverse lighting conditions. However, the process of obtaining polarization information causes it to be more sensitive to noise; serious noise reduces the quality of polarized images and subsequent performance in advanced visual tasks. Unfortunately, the flourishing low-light image enhancement methods applied to intensity images have not demonstrated satisfactory performance when transferred to polarized images. In this paper, we propose a low-light image enhancement paradigm based on the antagonistic properties of polarization parameters. Furthermore, we develop a dual-branch network that relies on a gradient residual dense feature extraction module (GRD) designed for polarized image characteristics and polarization loss, effectively avoiding noise introduced during the direct amplification of brightness, and capable of restoring target contour details. To facilitate a data-driven learning method, we propose a simulation method for underwater low-light polarized images. Extensive experimental results on real-world datasets demonstrate the effectiveness of our proposed approach and its superiority against other state-of-the-art methods.

An Apple Detection and Localization Method for Automated Harvesting under Adverse Light Conditions

The use of automation technology in agriculture has become particularly important as global agriculture is challenged by labor shortages and efficiency gains. The automated process for harvesting apples, an important agricultural product, relies on efficient and accurate detection and localization technology to ensure the quality and quantity of production. Adverse lighting conditions can significantly reduce the accuracy of fruit detection and localization in automated apple harvesting. Based on deep-learning techniques, this study aims to develop an accurate fruit detection and localization method under adverse light conditions. This paper explores the LE-YOLO model for accurate and robust apple detection and localization. The traditional YOLOv5 network was enhanced by adding an image enhancement module and an attention mechanism. Additionally, the loss function was improved to enhance detection performance. Secondly, the enhanced network was integrated with a binocular camera to achieve precise apple localization even under adverse lighting conditions. This was accomplished by calculating the 3D coordinates of feature points using the binocular localization principle. Finally, detection and localization experiments were conducted on the established dataset of apples under adverse lighting conditions. The experimental results indicate that LE-YOLO achieves higher accuracy in detection and localization compared to other target detection models. This demonstrates that LE-YOLO is more competitive in apple detection and localization under adverse light conditions. Compared to traditional manual and general automated harvesting, our method enables automated work under various adverse light conditions, significantly improving harvesting efficiency, reducing labor costs, and providing a feasible solution for automation in the field of apple harvesting.

SEEK+: Securing vehicle GPS via a sequential dashcam-based vehicle localization framework

Nowadays, the Global Positioning System (GPS) plays an critical role in providing navigational services for transportation and a variety of other location-dependent applications. However, the emergent threat of GPS spoofing attacks compromises the safety and reliability of these systems. In response, this study introduces a cutting-edge computer vision-based methodology, the SEquential dashcam-based vEhicle localization frameworK Plus (SEEK+), designed to counteract GPS spoofing. By analyzing dashcam footage to ascertain a vehicle’s actual location, SEEK+ scrutinizes the authenticity of reported GPS data, effectively identifying spoofing incidents. The application of dashcam imagery for localization, however, presents inherent obstacles, such as adverse lighting and weather conditions, seasonal and temporal image variations, obstructions within the camera’s field of view, and fluctuating vehicle velocities. To overcome these issues, SEEK+ integrates innovative strategies within its framework, demonstrating superior efficacy over existing approaches with a notable detection accuracy rate of up to 94%.

ECG Electrode Localization: 3D DS Camera System for Use in Diverse Clinical Environments

Models of the human body representing digital twins of patients have attracted increasing interest in clinical research for the delivery of personalized diagnoses and treatments to patients. For example, noninvasive cardiac imaging models are used to localize the origin of cardiac arrhythmias and myocardial infarctions. The precise knowledge of a few hundred electrocardiogram (ECG) electrode positions is essential for their diagnostic value. Smaller positional errors are obtained when extracting the sensor positions, along with the anatomical information, for example, from X-ray Computed Tomography (CT) slices. Alternatively, the amount of ionizing radiation the patient is exposed to can be reduced by manually pointing a magnetic digitizer probe one by one to each sensor. An experienced user requires at least 15 min. to perform a precise measurement. Therefore, a 3D depth-sensing camera system was developed that can be operated under adverse lighting conditions and limited space, as encountered in clinical settings. The camera was used to record the positions of 67 electrodes attached to a patient’s chest. These deviate, on average, by 2.0 mm ±1.5 mm from manually placed markers on the individual 3D views. This demonstrates that the system provides reasonable positional precision even when operated within clinical environments.

Event Camera and LiDAR based Human Tracking for Adverse Lighting Conditions in Subterranean Environments

In this article, we propose a novel LiDAR and event camera fusion modality for subterranean (SubT) environments for fast and precise object and human detection in a wide variety of adverse lighting conditions, such as low or no light, high-contrast zones and in the presence of blinding light sources. In the proposed approach, information from the event camera and LiDAR are fused to localize a human or an object-of-interest in a robot's local frame. The local detection is then transformed into the inertial frame and used to set references for a Nonlinear Model Predictive Controller (NMPC) for reactive tracking of humans or objects in SubT environments. The proposed novel fusion uses intensity filtering and K-means clustering on the LiDAR point cloud and frequency filtering and connectivity clustering on the events induced in an event camera by the returning LiDAR beams. The centroids of the clusters in the event camera and LiDAR streams are then paired to localize reflective markers present on safety vests and signs in SubT environments. The efficacy of the proposed scheme has been experimentally validated in a real SubT environment (a mine) with a Pioneer 3AT mobile robot. The experimental results show real-time performance for human detection and the NMPC-based controller allows for reactive tracking of a human or object of interest, even in complete darkness.

MTANet: Multitask-Aware Network With Hierarchical Multimodal Fusion for RGB-T Urban Scene Understanding

Understanding urban scenes is a fundamental ability requirement for assisted driving and autonomous vehicles. Most of the available urban scene understanding methods use red-green-blue (RGB) images; however, their segmentation performances are prone to degradation under adverse lighting conditions. Recently, many effective artificial neural networks have been presented for urban scene understanding and have shown that incorporating RGB and thermal (RGB-T) images can improve segmentation accuracy even under unsatisfactory lighting conditions. However, the potential of multimodal feature fusion has not been fully exploited because operations such as simply concatenating the RGB and thermal features or averaging their maps have been adopted. To improve the fusion of multimodal features and the segmentation accuracy, we propose a multitask-aware network (MTANet) with hierarchical multimodal fusion (multiscale fusion strategy) for RGB-T urban scene understanding. We developed a hierarchical multimodal fusion module to enhance feature fusion and built a high-level semantic module to extract semantic information for merging with coarse features at various abstraction levels. Using the multilevel fusion module, we exploited low-, mid-, and high-level fusion to improve segmentation accuracy. The multitask module uses boundary, binary, and semantic supervision to optimize the MTANet parameters. Extensive experiments were performed on two benchmark RGB-T datasets to verify the improved performance of the proposed MTANet compared with state-of-the-art methods. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>

Mask-guided modality difference reduction network for RGB-T semantic segmentation

By exploiting the complementary information of RGB modality and thermal modality, RGB-thermal (RGB-T) semantic segmentation is robust to adverse lighting conditions. When fusing features from RGB images and thermal images, the existing methods design different feature fusion strategies, but most of these methods overlook the modality differences caused by different imaging mechanisms. This may result in insufficient usage of complementary information. To address this issue, we propose a novel Mask-guided Modality Difference Reduction Network (MMDRNet), where the mask is utilized in the image reconstruction to ensure that the modality discrepancy within foreground regions is minimized. Doing so enables the generation of more discriminative representations for foreground pixels, thus facilitating the segmentation task. On top of this, we present a Dynamic Task Balance (DTB) method to balance the modality difference reduction task and semantic segmentation task dynamically. The experimental results on the MFNet dataset and the PST900 dataset demonstrate the superiority of the proposed mask-guided modality difference reduction strategy and the effectiveness of the DTB method.

ROBUST AND SCALABLE REAL-TIME VEHICLE CLASSIFICATION AND TRACKING: A CASE STUDY OF THAILAND

Abstract. An accurate detection, classification, and tracking of vehicles are highly important for intelligent transport systems (ITS) and road maintenance. In recent years, the deep learning (DL)-based approach is highly regarded for real-time vehicle classification from surveillance cameras. However, the practical implementation of such an approach is affected by the adverse lighting conditions and positioning of the cameras. In this research, we develop a DL-based method for near real-time multi-vehicle counting, classifying, and tracking on individual lanes of the road. First, we train a DL network of the You Only Look Once (YOLO) family on a custom dataset that we have curated. The dataset consists of nearly 30000 training samples to classify the vehicles into seven classes, which is more than in the existing benchmark datasets. Second, we fine-tune the trained model into another small dataset collected from the surveillance cameras that are used during the implementation process. Third, we connect the trained model to a tracking algorithm that we have developed to produce a per-lane report with the calculation of the speed and mobility of the vehicles. We test the robustness of the system on different faces of the vehicles and in adverse lighting conditions. The overall accuracy (OA) of classification ranges from 91% to 99% in four faces of vehicles (back, front, driver side, and passenger side). Similarly, in an experiment on adverse lighting conditions, OA of 93.7% and 99.6% is observed in a noisy and clear lighting conditions respectively. The implications of these results will assist in road maintenance with spatial information management and sensing for intelligent transport planning.

Trunk detection in tree crops using RGB-D images for structure-based ICM-SLAM

Agricultural environments with tree plantations usually present a regular structure that can be used by SLAM systems to improve self-location, and therefore, facilitate the autonomous navigation. In this context, tree trunks are natural landmarks that can be used to incorporate the environment structure into the problem modeling. This article presents a trunk detector solely based on RGB-D data obtained from a frontal-view stereo camera, and a SLAM system that incorporates the regular tree distribution of these crops. The trunk detector can be adapted to similar agricultural environments because its parameters have specific geometric meanings, which differentiates it from black box-type procedures. The structure-based SLAM system has theoretical and practical advantages over the well-known SLAM procedures in the mentioned context. This proposal can be executed on-line and is experimentally tested with databases obtained from a challenging agricultural environment. Results show a good performance and robustness when the database is spatially or temporally subsampled, even under adverse lighting conditions.

Lightweight Hardware Architecture for Object Detection in Driver Assistance Systems

Object detection on hardware platforms plays a very significant role in developing driver assistance systems (DASs) with limited computational resources. Object detection for DAS is a multiclass detection problem that involves detecting various objects like cars, auto, traffic lights, bicycles, pedestrians, etc. DAS also requires accuracy, speed, and sensitivity for detecting these objects in various challenging conditions. The lighting and weather conditions pose a serious challenge for accurate object detection for DAS. This paper proposes a speed-efficient and lightweight fully convolutional neural network (CNN) architecture for object detection in adverse rainy conditions. The proposed architecture uses a CNN-based deraining network with a custom SSIM loss function in the object detection pipeline, which can give an accurate performance using limited computational and memory resources. The object detection architecture contains some architectural modifications to the existing single shot multibox detector (SSD) architecture to make it more hardware efficient and improve accuracy on small objects. It uses a trainable color transformation module using [Formula: see text] convolutions for handling the adverse lighting conditions encountered in DAS. The architecture uses feature fusion and the dilated convolution approach to enhance the accuracy of the proposed architecture on small objects. The datasets available for object detection in DAS are very imbalanced with cars as a predominant object. The class weight penalization technique is used to improve the performance of the architecture on scarcely present objects. The performance of the architecture is evaluated on well-known datasets like Kitti, Udacity, Indian Driving Dataset (IDD), and DAWN. The architecture achieves satisfactory performance in terms of mean average precision (mAP) and detection time on all these datasets. It requires three times fewer hardware resources compared to existing architectures. The lightweight nature of the proposed architecture and modification of CNN architecture with TensorRT allow the efficient implementation on the jetson nanohardware platform for prototyping, which can be integrated with other intelligent transportation systems.

A framework for mode classification in multimodal environments using radar-based sensors

Monitoring traffic at locations where bicyclists and pedestrians are present and studying their interactions with motorized vehicles has the potential to reveal underlying crash mechanisms and, in turn, guide the implementation of suitable countermeasures. Previous research has demonstrated the capabilities of vision-based systems in traffic monitoring; however, as these systems rely on video camera data, they remain constrained in adverse lighting and weather conditions. Although radar-based sensors can be used in traffic monitoring, they have not been tested in multimodal environments. To bridge this gap a novel framework to classify trajectories recorded by radar-based sensors in multimodal traffic environments is developed. The Support Vector Machine is used as the classifier and the following aspects allow to develop a robust, flexible, and transferable classification framework that can be applied in various traffic scenes. The SVM employs (1) a speed and length or speed and acceleration feature vector, (2) trajectory normalization scheme (i.e., select multiple measurements per trajectory), (3) training sample balancing strategy, and (4) cross-validation strategy. The framework is tested and validated using data from two different multimodal intersections. The results suggest that the mode type for every trajectory can be predicted with 95% accuracy using ten speed measurements and the vehicle’s mean length while accounting for the unequal number of observations per class. While these results are subject to change, i.e., both SVM’s input (feature vector and/or balancing approach) and performance may vary across different traffic scenes, the proposed framework is transferable and flexible.

Uncertainty Evaluation of Object Detection Algorithms for Autonomous Vehicles

The safety of the intended functionality (SOTIF) has become one of the hottest topics in the field of autonomous driving. However, no testing and evaluating system for SOTIF performance has been proposed yet. Therefore, this paper proposes a framework based on the advanced You Only Look Once (YOLO) algorithm and the mean Average Precision (mAP) method to evaluate the object detection performance of the camera under SOTIF-related scenarios. First, a dataset is established, which contains road images with extreme weather and adverse lighting conditions. Second, the Monte Carlo dropout (MCD) method is used to analyze the uncertainty of the algorithm and draw the uncertainty region of the predicted bounding box. Then, the confidence of the algorithm is calibrated based on uncertainty results so that the average confidence after calibration can better reflect the real accuracy. The uncertainty results and the calibrated confidence are proposed to be used for online risk identification. Finally, the confusion matrix is extended according to the several possible mistakes that the object detection algorithm may make, and then the mAP is calculated as an index for offline evaluation and comparison. This paper offers suggestions to apply the MCD method to complex object detection algorithms and to find the relationship between the uncertainty and the confidence of the algorithm. The experimental results verified by specific SOTIF scenarios proof the feasibility and effectiveness of the proposed uncertainty acquisition approach for object detection algorithm, which provides potential practical implementation chance to address perceptual related SOTIF risk for autonomous vehicles.

Implementation of the electronic sector of the satellite camera and image contrast enhancement

The purpose of this paper is to design and implement the electronic sector of satellite camera on the basis of systematic calculations as well as presenting the general and detailed block diagram. The main parts of the designed camera consist of four units named “optics, detector, processors, and memory” that can work in one of three modes: real-time, storage and storage-send. Verification of the simulation and practical results are shown completely by receiving images. According to the conditions of the satellite imaging, in most cases, there is a need to improve the image quality. A very important feature in the satellite images is contrast enhancement. In the following, a powerful contrast enhancement algorithm is proposed based on histogram equalization method called the “entropy-based triple dynamic clipped histogram equalization” (ETDCHE). One of the strengths of this paper is using images with diverse variations in brightness. Such that by producing clear images and by preserving maximum details this overcomes the adverse lighting conditions and leads to natural enhancement of the output images. Also, the performance assessment of the proposed method in terms of the average information content reflects the considerable superiority of the proposed algorithm compared to the previously presented methods based on histogram equalization.

Dronument: System for Reliable Deployment of Micro Aerial Vehicles in Dark Areas of Large Historical Monuments

This letter presents a self-contained system for robust deployment of autonomous aerial vehicles in environments without access to global navigation systems and with limited lighting conditions. The proposed system, application-tailored for documentation in dark areas of large historical monuments, uses a unique and reliable aerial platform with a multi-modal lightweight sensory setup to acquire data in human-restricted areas with adverse lighting conditions, especially in areas that are high above the ground. The introduced localization method relies on an easy-to-obtain 3-D point cloud of a historical building, while it copes with a lack of visible light by fusing active laser-based sensors. The approach does not rely on any external localization, or on a preset motion-capture system. This enables fast deployment in the interiors of investigated structures while being computationally undemanding enough to process data online, onboard an MAV equipped with ordinary processing resources. The reliability of the system is analyzed, is quantitatively evaluated on a set of aerial trajectories performed inside a real-world church, and is deployed onto the aerial platform in the position control feedback loop to demonstrate the reliability of the system in the safety-critical application of historical monuments documentation.

Deep HDR Imaging via A Non-local Network.

One of the most challenging problems in reconstructing a high dynamic range (HDR) image from multiple low dynamic range (LDR) inputs is the ghosting artifacts caused by the object motion across different inputs. When the object motion is slight, most existing methods can well suppress the ghosting artifacts through aligning LDR inputs based on optical flow or detecting anomalies among them. However, they often fail to produce satisfactory results in practice, since the real object motion can be very large. In this study, we present a novel deep framework, termed NHDRRnet, which adopts an alternative direction and attempts to remove ghosting artifacts by exploiting the non-local correlation in inputs. In NHDRRnet, we first adopt an Unet architecture to fuse all inputs and map the fusion results into a low-dimensional deep feature space. Then, we feed the resultant features into a novel global non-local module which reconstructs each pixel by weighted averaging all the other pixels using the weights determined by their correspondences. By doing this, the proposed NHDRRnet is able to adaptively select the useful information (e.g., which are not corrupted by large motions or adverse lighting conditions) in the whole deep feature space to accurately reconstruct each pixel. In addition, we also incorporate a triple-pass residual module to capture more powerful local features, which proves to be effective in further boosting the performance. Extensive experiments on three benchmark datasets demonstrate the superiority of the proposed NDHRnet in terms of suppressing the ghosting artifacts in HDR reconstruction, especially when the objects have large motions.

On the Safe Road Toward Autonomous Driving: Phase Noise Monitoring in Radar Sensors for Functional Safety Compliance

The first approaches to improve vehicle safety were so-called passive safety systems, which did not directly interfere with the driving process but protected the occupants during a crash. In contrast, the first assistance system was the antilock braking system (ABS) successfully introduced in the early 1970s. This active system was developed to avoid an accident by automatically intervening in the braking behavior of the car. At about the same time, the first automotive radar prototype was presented. Since the invention of this very unwieldy radar system, organizations all around the world spent significant efforts in pushing the development of automotive radar systems forward. Today, radar sensors together with ultrasound sensors, lidar, and cameras form the backbone of advanced driver assistant systems (ADASs) as well as autonomous driving (AD), which is in the prototype stage. In particular, because of their robustness against adverse lighting and weather conditions, radar sensors are considered a key technology for modern vehicle safety and comfort systems. Along with the trend toward higher automation, more cars will be equipped with radar sensors in the near future. Because ADASs directly influence the vehicle dynamics, new regulating functional safety (FuSa) requirements, such as the ISO 26262 standard, were introduced. These requirements are mandatory to protect the road users.

Dynamic texture recognition under adverse lighting and weather conditions for outdoor environments

Recognizing dynamic patterns based on visual processing is significant for many applications. In this paper dynamic texture recognition focuses on outdoor scenarios where a crisis event might occur (i.e. fire in a forest, floods/flooding etc.) Real outdoor scenes may include the objects with dynamic behaviour due to illumination, blurring, or weather conditions effects. Under bad weather conditions the imaging systems is degraded and produce low visibility images. In this work precipitation artefacts and lightning effects for dynamic texture analysis were studied. Experimental results show that the proposed method of weather and adverse lighting effects compensation is feasible and effective for videobased dynamic texture analysis under bad weather conditions.

A collaborative client participant fusion system for realistic remote conferences

Remote conferencing systems provide a shared environment where people in different locations can communicate and collaborate in real time. Currently, remote video conferencing systems present separate video images of the individual participants. To achieve a more realistic conference experience, we enhance video conferencing by integrating the remote images into a shared virtual environment. This paper proposes a collaborative client participant fusion system using a real-time foreground segmentation method. In each client system, the foreground pixels are extracted from the participant images using a feedback background modeling method. Because the segmentation results often contain noise and holes caused by adverse environmental lighting conditions and substandard camera resolution, a Markov Random Field model is applied in the morphological operations of dilation and erosion. This foreground segmentation refining process is implemented using graphics processing unit programming, to facilitate real-time image processing. Subsequently, segmented foreground pixels are transmitted to a server, which fuses the remote images of the participants into a shared virtual environment. The fused conference scene is represented by a realistic holographic projection.