Extreme Illumination Research Articles

A Novel Edge Detection Method of Blade With Multisupervision for Foreground–Background Confusion Caused by Extreme Illumination

A Novel Edge Detection Method of Blade With Multisupervision for Foreground–Background Confusion Caused by Extreme Illumination

Convolutional MLP orthogonal fusion of multiscale features for visual place recognition

Visual place recognition (VPR) involves obtaining robust image descriptors to cope with differences in camera viewpoints and drastic external environment changes. Utilizing multiscale features improves the robustness of image descriptors; however, existing methods neither exploit the multiscale features generated during feature extraction nor consider the feature redundancy problem when fusing multiscale information when image descriptors are enhanced. We propose a novel encoding strategy—convolutional multilayer perceptron orthogonal fusion of multiscale features (ConvMLP-OFMS)—for VPR. A ConvMLP is used to obtain robust and generalized global image descriptors and the multiscale features generated during feature extraction are used to enhance the global descriptors to cope with changes in the environment and viewpoints. Additionally, an attention mechanism is used to eliminate noise and redundant information. Compared to traditional methods that use tensor splicing for feature fusion, we introduced matrix orthogonal decomposition to eliminate redundant information. Experiments demonstrated that the proposed architecture outperformed NetVLAD, CosPlace, ConvAP, and other methods. On the Pittsburgh and MSLS datasets, which contained significant viewpoint and illumination variations, our method achieved 92.5% and 86.5% Recall@1, respectively. We also achieved good performances—80.6% and 43.2%—on the SPED and NordLand datasets, respectively, which have more extreme illumination and appearance variations.

Mineralogical Characterization of the Lunar South Polar Region: 1. The Artemis Exploration Zone

AbstractThe lunar south pole is a region of focused scientific and exploration interest, with several crewed and robotic missions to this region planned within the next decade. Understanding the mineralogy of the region is essential to inform landing site characterization and selection and provides the key context for interpreting samples and in situ observations. At high latitudes, extreme illumination conditions (high phase angles) can negatively impact the data quality of orbital instruments. This is especially true for passive near‐infrared spectrometers such as the Moon Mineralogy Mapper (M3) and the Kaguya Spectral Profiler, which measure the spectral properties of the surface using reflected sunlight. Using Moon Mineralogy Mapper data, we observed that the south polar region is associated with a detectable mafic signature consistent with the presence of pyroxenes. The strongest mafic signatures are associated with the South Pole—Aitken Basin, suggesting that impact melt and basin ejecta from the lower crust and upper mantle are present within this region. This observation is validated in several ways: (a) comparisons between M3 data acquired during different mission phases, (b) comparisons between multiple spectral parameters sensitive to the presence of mafic minerals, (c) comparisons between the north and south lunar polar regions, and (d) comparisons with publicly available Kaguya polar mineralogy maps and Lunar Prospector elemental abundances. We also investigate the nature of an anomalous high‐albedo region within 2–3° of the south pole observed in Lunar Orbiter Laser Altimeter reflectance data exhibiting a spatially conflicting apparent FeO abundance pattern between several data sets.

VERI-D: A new dataset and method for multi-camera vehicle re-identification of damaged cars under varying lighting conditions

Vehicle re-identification (V-ReID) is a critical task that aims to match the same vehicle across images from different camera viewpoints. The previous studies have leveraged attribute clues, such as color, model, and license plate, to enhance the V-ReID performance. However, these methods often lack effective interaction between the global–local features and the final V-ReID objective. Moreover, they do not address the challenging issues in real-world scenarios, such as high viewpoint variations, extreme illumination conditions, and car appearance changes (e.g., due to damage or wrong driving). We propose a novel framework to tackle these problems and advance the research in V-ReID, which can handle various types of car appearance changes and achieve robust V-ReID under varying lighting conditions. Our main contributions are as follows: (i) we propose a new Re-ID architecture named global–local self-attention network, which integrates local information into the feature learning process and enhances the feature representation for V-ReID and (ii) we introduce a novel damaged vehicle Re-ID dataset called VERI-D, which is the first publicly available dataset that focuses on this challenging yet practical scenario. The dataset contains both natural and synthetic images of damaged vehicles captured from multiple camera viewpoints and under different lighting conditions. (iii) We conduct extensive experiments on the VERI-D dataset and demonstrate the effectiveness of our approach in addressing the challenges associated with damaged vehicle re-identification. We also compare our method to several state-of-the-art V-ReID methods and show its superiority.

Genomic imprints of unparalleled growth.

Chlorella ohadii was isolated from desert biological soil crusts, one of the harshest habitats on Earth, and is emerging as an exciting new green model for studying growth, photosynthesis and metabolism under a wide range of conditions. Here, we compared the genome of C. ohadii, the fastest growing alga on record, to that of other green algae, to reveal the genomic imprints empowering its unparalleled growth rate and resistance to various stressors, including extreme illumination. This included the genome of its close relative, but slower growing and photodamage sensitive, C. sorokiniana UTEX 1663. A larger number of ribosome-encoding genes, high intron abundance, increased codon bias and unique genes potentially involved in metabolic flexibility and resistance to photodamage are all consistent with the faster growth of C. ohadii. Some of these characteristics highlight general trends in Chlorophyta and Chlorella spp. evolution, and others open new broad avenues for mechanistic exploration of their relationship with growth. This work entails a unique case study for the genomic adaptations and costs of exceptionally fast growth and sheds light on the genomic signatures of fast growth in photosynthetic cells. It also provides an important resource for future studies leveraging the unique properties of C. ohadii for photosynthesis and stress response research alongside their utilization for synthetic biology and biotechnology aims.

Neuromorphic imaging and classification with graph learning

Bio-inspired neuromorphic cameras asynchronously record pixel brightness changes and generate sparse event streams. They can capture dynamic scenes with little motion blur and more details in extreme illumination conditions. Due to the multidimensional address-event structure, most existing vision algorithms cannot properly handle asynchronous event streams. While several event representations and processing methods have been developed to address such an issue, they are typically driven by a large number of events, leading to substantial overheads in runtime and memory. In this paper, we propose a new graph representation of the event data and couple it with a Graph Transformer to perform accurate neuromorphic classification. Extensive experiments show that our approach leads to better results and excels at the challenging realistic situations where only a small number of events and limited computational resources are available, paving the way for neuromorphic applications embedded into mobile facilities.

Adaptive Feature Attention Module for Robust Visual–LiDAR Fusion-Based Object Detection in Adverse Weather Conditions

Object detection is one of the vital components used for autonomous navigation in dynamic environments. Camera and lidar sensors have been widely used for efficient object detection by mobile robots. However, they suffer from adverse weather conditions in operating environments such as sun, fog, snow, and extreme illumination changes from day to night. The sensor fusion of camera and lidar data helps to enhance the overall performance of an object detection network. However, the diverse distribution of training data makes the efficient learning of the network a challenging task. To address this challenge, we systematically study the existing visual and lidar features based on object detection methods and propose an adaptive feature attention module (AFAM) for robust multisensory data fusion-based object detection in outdoor dynamic environments. Given the camera and lidar features extracted from the intermediate layers of EfficientNet backbones, the AFAM computes the uncertainty among the two modalities and adaptively refines visual and lidar features via attention along the channel and the spatial axis. The AFAM integrated with the EfficientDet performs the adaptive recalibration and fusion of visual lidar features by filtering noise and extracting discriminative features for an object detection network under specific environmental conditions. We evaluate the AFAM on a benchmark dataset exhibiting weather and light variations. The experimental results demonstrate that the AFAM significantly enhances the overall detection accuracy of an object detection network.

Contrastive fusion representation learning for foreground object detection

This paper investigates the problem of an effective and robust fusion representation method for foreground moving object detection. Many fusion representation learning approaches pay attention to the similarity measurement between the fusion results and source images in texture details and pixel intensity, ignoring the harmful information, e.g., noise, blur, and extreme illumination. Therefore, the aggregated features of infrared and visible images will introduce much harmful information, affecting the model performance of downstream visual tasks. This paper tackles these problems by proposing a contrastive fusion representation learning method for the foreground moving object detection task, which consists of two major modules: the upstream fusion representation module (FRM) and the downstream foreground moving object detection module (FODM). Unlike the traditional fusion optimization mechanism, the former aims to extract valuable features and reject harmful features via the maximum mutual information theory. The latter is a siamese convolutional neural network to detect foreground moving objects by aggregating the time-sequence images generated by FRM. Experimental results and comparisons with the state-of-the-art on three public datasets (i.e., TNO, MF, and cross-modal FOD dataset of infrared and visible images), validate the effectiveness, robustness, and overall superiority of the proposed contrast fusion representation learning method. Concisely, our contrastive fusion representation learning has gained 53.9%,43.2%,46.4%,52.3%,2.2%,87.1%,3.5% on EI, SF, DF, AG, MI, and Nabf metrics compared with the best competitors.

Extracting vegetation information from high dynamic range images with shadows: A comparison between deep learning and threshold methods

Due to factors such as light angle and vegetation density, shadows can exist in ground vegetation images; this can lead to highly varying natural illumination in the images, which greatly limits the accuracy of fractional vegetation cover (FVC) estimation. While high dynamic range (HDR) images can reduce the extreme illumination contrast, the characteristic vegetation information in shadow areas is often lost. At present, most threshold methods do not fully utilize the spatial features of vegetation, and the results of vegetation classification under complex shadow conditions are associated with large errors. Thus, two deep learning methods (the HDR U-Net method and HDR U-Net++ method) are proposed in this study based on fully convolutional neural networks to extract vegetation information from HDR images under shadow conditions. These methods are compared with threshold methods. For full HDR images, the kappa coefficient and mean intersection over union (MIoU) of the HDR U-Net method and HDR U-Net++ method are 0.926, 0.931, and 0.874, 0.887, respectively. The FVC estimation accuracy and vegetation segmentation effect of these two deep learning methods are better than those of the three threshold methods considered. In addition, compared with normal exposure (NE) images, camera-based HDR images improve the extraction accuracy of deep learning methods and threshold methods for vegetation under shadow conditions. The results of this study suggest that deep learning methods are not affected by the threshold determination strategy selected and can more completely extract vegetation feature information from HDR images. The classification effect of vegetation under shadow conditions is effectively improved, and the vegetation segmentation and FVC estimation results are more precise. The combination of HDR images and deep learning methods can be applied to field scenes with complex lighting, which is beneficial to improve the ground data verification of remote sensing FVC products.

3-D Facial Feature Reconstruction and Learning Network for Facial Expression Recognition in the Wild

Facial expression recognition (FER) in the wild unavoidably suffers from the effects of face posture, illumination, and partial occlusion. In this article, we attempt to alleviate the above negative effects and improve the performance of FER in the wild based on 3-D face feature reconstruction and learning. Three-dimensional face reconstruction not only can effectively make up for the facial apparent information missing inform a 2-D face images but can also extract accurate 3-D facial geometric information in self-occlusion and extreme illumination scenarios. Therefore, we propose a novel end-to-end trainable 3-D face feature reconstruction and learning network (3-DF-RLN) is proposed for FER in the wild. In 3-DF-RLN, the 2-D implicitly frontalized face apparent data and 3-D facial landmarks are reconstructed by a 3-D face reconstruction module and input to two feature extraction pathways. The appearance pathway learns apparent features from the reconstructed 2-D face apparent data using a convolutional neural network. The geometry pathway learns the geometric features from the reconstructed 3-D facial landmarks using a graph convolutional network. Finally, FER is achieved via the fusion of the two pathways. Extensive experiments were conducted to evaluate the proposed method with three benchmark databases, including Multi-PIE, RAF-DB, and AffectNet. The results show that the proposed 3-DF-RLN model has better FER performance, both in the lab and in the wild. In addition, the face graph from the geometry pathway reveals the correlations between facial landmarks in FER.

Appearance and geometry transformer for facial expression recognition in the wild

Facial expression recognition (FER) in the wild is considered to be one of the most challenging tasks in computer vision because it requires identifying the categories of expressions presented by the human face under adverse conditions, such as dynamic poses, extreme illuminations, and partial occlusion. In this paper, we propose a model called the appearance and geometry transformer (AGT), which is a self-attention-based deep neural network, to improve the accuracy of recognition of FER in the wild. The AGT is designed to simultaneously perform feature extraction and fusion on heterogeneous data, including images and graphs, by using two transformer pathways. A pre-trained DeiT model is introduced to the image transformer pathway to explore the most discriminative facial regions. A multi-scale graph transformer network consisting of a self-attention module based on several multi-scale graphs is presented in the graph transformer pathway to adaptively extract features from the facial graph. A distillation mechanism is employed in the AGT to improve training in case of a limited number of training samples. We evaluated the proposed method on two benchmark databases: FERplus and RAF-DB. The results of comprehensive experiments on these two databases show that the proposed model can achieve comparable results to state-of-the-art methods of FER in the wild.

Influence of the threshold settings of photon counting detectors on the observed speckle contrast in coherent scattering experiments

Speckle-based experiments, in particular X-ray Photon Correlation Spectroscopy (XPCS), are among the ones that benefit most from the development of next generation light sources. The key quantity that determines whether or not it will be possible to perform an XPCS experiment is the speckle contrast β, which measures the visibility of the speckle pattern. The speckle contrast is influenced by geometrical conditions and light-source properties [1], and recently it has been noticed that artifacts in photon-counting detectors strongly interfere with the determination of the real value of β when the number of detected photons per frame is extremely low [2]. Here we report how the threshold setting of a photon counting detector, an EIGER X 4M, affects the detected contrast even at non extreme illumination conditions (0.01-0.05 photons/pixel/second). We found that by increasing the threshold value, not only leads to the expected drop in detected intensity, but also to a significant increase in the value of β. Additionally we will present an in-operando situation in which the higher threshold helps critically to increase the signal-to-noise ratio of the measured scattering pattern resulting in a five-fold increment of the speckle contrast.

Efficient and Robust Shadowing Determination for Mesh-Based Physical Optics Analysis

Visibility status determination for every mesh element/edge with respect to an illuminating source is routinely required in mesh-based physical optics (PO) methods, where the scatterer is represented by subwavelength sized elements. This article presents an optimally efficient shadowing determination algorithm for both plane wave and point source (Hertzian dipole) illumination of large-scale triangle meshes. It is based on a recursive multilevel field-of-view (image space) subdivision buffer approach, whereby a point’s status is determined by only checking against a small subset of elements. It is error-free and formulated to maintain efficiency for inhomogeneous meshes and grazing incidence, which result in concentration of projected elements in small areas of the buffer plane. A key step is to separate the grazing and nongrazing parts of the mesh. Very challenging test cases affirm the algorithm’s expected optimal and robust linear computational cost scaling of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mathcal {O}(N)$ </tex-math></inline-formula> , where <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$N$ </tex-math></inline-formula> denotes the number of mesh elements. For fixed <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$N$ </tex-math></inline-formula> , almost constant runtimes under extreme illumination and mesh variations, further demonstrate robustness.

Topographic Correction of the SELENE MI Images with the LOLA DEM around Shackleton Crater

Due to the complicated terrain and extreme illumination condition in the lunar south polar region, topography has an impact on the reflectance of the area. This may result in the misinterpretation of the properties of the lunar surface. Therefore, an image obtained across Shackleton Crater by the Multi-band Imager (MI) onboard the first Japanese lunar orbiter SELENE (SELenological and ENgineering Explorer) was utilized to evaluate the topographic effect on the reflectance of the polar region. Three methods—i.e., b correction, C correction, and Minnaert correction—were applied to the MI image to reduce the topographic effect. It was found that the reflectance result of C correction suffers from overcorrection. The topographic effect is enlarged, rather than suppressed, in the reflectance derived with Minnaert correction. Comparatively, b correction is the more appropriate method for the topographic correction of the MI image across the study area. The reflectance of the sunlit slope of the crater wall in the study area decreased by ~60%. The reflectance of the area outside of the crater, with a gentler slope compared to the crater wall, decreased by ~20%. Meanwhile, according to the correlation of the cosine of the local solar incidence angle and the corrected reflectance, the topographic effect was seemingly not completely eliminated in the MI image. However, by analyzing the spectral absorption features of the 1250 nm band, we can attribute this “residual” effect to the different compositions inside and outside of Shackleton Crater, most likely caused by a high concentration of plagioclase. Topographic correction of the MI images over NASA candidate landing regions was also conducted. The results suggest that the topographic effect can contribute significantly to the reflectance of the sunlit slopes, and should not be neglected in the analysis of the reflectance images. This study highlights the topographic impact on the reflectance of the lunar south pole’s surface. In addition, our results also suggest that the compositional differences under various terrains should be considered in the evaluation of the topographic correction of specific regions, such as Shackleton Crater’s inner walls.

PIAFusion: A progressive infrared and visible image fusion network based on illumination aware

Infrared and visible image fusion aims to synthesize a single fused image containing salient targets and abundant texture details even under extreme illumination conditions. However, existing image fusion algorithms fail to take the illumination factor into account in the modeling process. In this paper, we propose a progressive image fusion network based on illumination-aware, termed as PIAFusion, which adaptively maintains the intensity distribution of salient targets and preserves texture information in the background. Specifically, we design an illumination-aware sub-network to estimate the illumination distribution and calculate the illumination probability. Moreover, we utilize the illumination probability to construct an illumination-aware loss to guide the training of the fusion network. The cross-modality differential aware fusion module and halfway fusion strategy completely integrate common and complementary information under the constraint of illumination-aware loss. In addition, a new benchmark dataset for infrared and visible image fusion, i.e., Multi-Spectral Road Scenarios (available at https://github.com/Linfeng-Tang/MSRS), is released to support network training and comprehensive evaluation. Extensive experiments demonstrate the superiority of our method over state-of-the-art alternatives in terms of target maintenance and texture preservation. Particularly, our progressive fusion framework could round-the-clock integrate meaningful information from source images according to illumination conditions. Furthermore, the application to semantic segmentation demonstrates the potential of our PIAFusion for high-level vision tasks. Our codes will be available at https://github.com/Linfeng-Tang/PIAFusion.

Automated License Plate Recognition for Resource-Constrained Environments.

The incorporation of deep-learning techniques in embedded systems has enhanced the capabilities of edge computing to a great extent. However, most of these solutions rely on high-end hardware and often require a high processing capacity, which cannot be achieved with resource-constrained edge computing. This study presents a novel approach and a proof of concept for a hardware-efficient automated license plate recognition system for a constrained environment with limited resources. The proposed solution is purely implemented for low-resource edge devices and performed well for extreme illumination changes such as day and nighttime. The generalisability of the proposed models has been achieved using a novel set of neural networks for different hardware configurations based on the computational capabilities and low cost. The accuracy, energy efficiency, communication, and computational latency of the proposed models are validated using different license plate datasets in the daytime and nighttime and in real time. Meanwhile, the results obtained from the proposed study have shown competitive performance to the state-of-the-art server-grade hardware solutions as well.

Towards real-time tracking and counting of seedlings with a one-stage detector and optical flow

The population of crop seedlings is important for breeders and growers to evaluate the emergence rate of different cultivars and the necessity of replanting, but manual counting of plant seedlings is time-consuming and tedious. Building upon our prior work, we advanced the cotton seedling tracking method by incorporating a one-stage object detection deep neural network and optical flow to improve tracking speed and counting accuracy. Videos of cotton seedlings were captured using consumer-grade video cameras from the top view. You Only Look Once Version 4 (YOLOv4), a one-stage object detection network, was trained to detect cotton seedlings in each frame and to generate bounding boxes. To associate the same seedlings between adjacent frames, an optical flow-based tracking method was adopted to estimate camera motions. By comparing the positions of bounding boxes predicted by optical flow and detected by the YOLOv4 network in the same frame, the number of cotton seedlings was updated. The trained YOLOv4 model achieved high accuracy under conditions of occlusions, blurry images, complex backgrounds, and extreme illuminations. The F1 score of the final detection model was 0.98 and the average precision was 99.12%. Important tracking metrics were compared to evaluate the tracking performance. The Multiple-Object Tracking Accuracy (MOTA) and ID switch of the proposed tracking method were 72.8% and 0.1%, respectively. Counting results showed that the relative error of all testing videos was 3.13%. Compared with the Kalman filter and particle filter-based methods, our optical flow-based method generated fewer errors on testing videos because of higher accuracy of motion estimation. Compared with our previous work, the RMSE of the optical flow-based method decreased by 0.54 and the counting speed increased from 2.5 to 10.8 frames per second. The counting speed can reach 16.6 frames per second if the input resolution was reduced to 1280 × 720 pixels with an only 0.45% reduction in counting accuracy. The proposed method provides an automatic and near real-time tracking approach for counting of multiple cotton seedlings in video frames with improved speed and accuracy, which will benefit plant breeding and precision crop management.

Direct Near-Infrared-Depth Visual SLAM With Active Lighting

Since visible cameras rely on ideal illumination to provide adequate environment information, visual simultaneous localization and mapping (SLAM) under extreme illumination remains a challenge. Therefore, we propose a direct near-infrared-depth visual SLAM method with an active near-infrared (NIR) light source. The NIR light source provides necessary lighting conditions without interference to human activities. The proposed method utilizes a direct bundle adjustment method jointly fusing near-infrared and depth images to optimize camera motion. In the front-end tracking, the albedo transformed images and depth maps are used in a direct image alignment. During the back-end optimization, the direct bundle adjustment method exploits the albedo-consistency clues to minimize the geometry and albedo residuals. The indoor experiments show that the proposed method can provide robust performances without being affected by visible illumination. Compared with leading SLAM methods using active light source, the proposed method still offers competitive and robust results. The results also show that NIR-based direct SLAM outperforms the same method based on visible light, indicating that the NIR-based implementation is a potential solution for robust pose estimation indoors.

Practical Face Reconstruction via Differentiable Ray Tracing

AbstractWe present a differentiable ray‐tracing based novel face reconstruction approach where scene attributes – 3D geometry, reflectance (diffuse, specular and roughness), pose, camera parameters, and scene illumination – are estimated from unconstrained monocular images. The proposed method models scene illumination via a novel, parameterized virtual light stage, which in‐conjunction with differentiable ray‐tracing, introduces a coarse‐to‐fine optimization formulation for face reconstruction. Our method can not only handle unconstrained illumination and self‐shadows conditions, but also estimates diffuse and specular albedos. To estimate the face attributes consistently and with practical semantics, a two‐stage optimization strategy systematically uses a subset of parametric attributes, where subsequent attribute estimations factor those previously estimated. For example, self‐shadows estimated during the first stage, later prevent its baking into the personalized diffuse and specular albedos in the second stage. We show the efficacy of our approach in several real‐world scenarios, where face attributes can be estimated even under extreme illumination conditions. Ablation studies, analyses and comparisons against several recent state‐of‐the‐art methods show improved accuracy and versatility of our approach. With consistent face attributes reconstruction, our method leads to several style – illumination, albedo, self‐shadow – edit and transfer applications, as discussed in the paper.

Using geometric constraints to improve performance of image classifiers for automatic segmentation of traffic signs

Automatic detection and recognition of traffic signs from images is an important topic in many applications. At first, we segmented the images using a classification algorithm to delineate the areas where the signs are more likely to be found. In this regard, shadows, objects having similar colours, and extreme illumination changes can significantly affect the segmentation results. We propose a new shape-based algorithm to improve the accuracy of the segmentation. The algorithm works by incorporating the sign geometry to filter out the wrong pixels from the classification results. We performed several tests to compare the performance of our algorithm against those obtained by popular techniques such as Support Vector Machine (SVM), K-Means, and K-Nearest Neighbours. In these tests, to overcome the unwanted illumination effects, the images are transformed into colour spaces Hue, Saturation, and Intensity, YUV, normalized red green blue, and Gaussian. Among the traditional techniques used in this study, the best results were obtained with SVM applied to the images transformed into the Gaussian colour space. The comparison results also suggested that by adding the geometric constraints proposed in this study, the quality of sign image segmentation is improved by 10%–25%. We also comparted the SVM classifier enhanced by incorporating the geometry of signs with a U-Shaped deep learning algorithm. Results suggested the performance of both techniques is very close. Perhaps the deep learning results could be improved if a more comprehensive data set is provided.