Complex Illumination Conditions Research Articles

Flexible retinomorphic vision sensors with scotopic and photopic adaptation for a fully flexible neuromorphic machine vision system

AbstractBioinspired neuromorphic machine vision system (NMVS) that integrates retinomorphic sensing and neuromorphic computing into one monolithic system is regarded as the most promising architecture for visual perception. However, the large intensity range of natural lights and complex illumination conditions in actual scenarios always require the NMVS to dynamically adjust its sensitivity according to the environmental conditions, just like the visual adaptation function of the human retina. Although some opto‐sensors with scotopic or photopic adaption have been developed, NMVSs, especially fully flexible NMVSs, with both scotopic and photopic adaptation functions are rarely reported. Here we propose an ion‐modulation strategy to dynamically adjust the photosensitivity and time‐varying activation/inhibition characteristics depending on the illumination conditions, and develop a flexible ion‐modulated phototransistor array based on MoS2/graphdiyne heterostructure, which can execute both retinomorphic sensing and neuromorphic computing. By controlling the intercalated Li+ ions in graphdiyne, both scotopic and photopic adaptation functions are demonstrated successfully. A fully flexible NMVS consisting of front‐end retinomorphic vision sensors and a back‐end convolutional neural network is constructed based on the as‐fabricated 28 × 28 device array, demonstrating quite high recognition accuracies for both dim and bright images and robust flexibility. This effort for fully flexible and monolithic NMVS paves the way for its applications in wearable scenarios.

Advanced underwater image restoration in complex illumination conditions

Underwater image restoration has been a challenging problem for decades since the advent of underwater photography. Most solutions focus on shallow water scenarios, where the scene is uniformly illuminated by the sunlight. However, the vast majority of uncharted underwater terrain is located beyond 200 meters depth where natural light is scarce and artificial illumination is needed. In such cases, light sources co-moving with the camera, dynamically change the scene appearance, which make shallow water restoration methods inadequate. In particular for multi-light source systems (composed of dozens of LEDs nowadays), calibrating each light is time-consuming, error-prone and tedious, and we observe that only the integrated illumination within the viewing volume of the camera is critical, rather than the individual light sources. The key idea of this paper is therefore to exploit the appearance changes of objects or the seafloor, when traversing the viewing frustum of the camera. Through new constraints assuming Lambertian surfaces, corresponding image pixels constrain the light field in front of the camera, and for each voxel a signal factor and a backscatter value are stored in a volumetric grid that can be used for very efficient image restoration of camera-light platforms, which facilitates consistently texturing large 3D models and maps that would otherwise be dominated by lighting and medium artifacts. To validate the effectiveness of our approach, we conducted extensive experiments on simulated and real-world datasets. The results of these experiments demonstrate the robustness of our approach in restoring the true albedo of objects, while mitigating the influence of lighting and medium effects. Furthermore, we demonstrate our approach can be readily extended to other scenarios, including in-air imaging with artificial illumination or other similar cases.

I2-FaçadeNet: An Illumination-invariant Façade Recognition Network Leveraging Sparsely Gated Mixture of Multi-color Space Experts for Aerial Oblique Imagery

Façade image recognition under complex illumination conditions is crucial for various applications, including urban three-dimensional modeling and building identification. Existing methods relying solely on Red-Green-Blue (RGB) images are prone to texture ambiguity in complex illumination environments. Furthermore, façades display varying orientations and camera viewing angles, resulting in performance issues within the RGB color space. In this study, we introduce an illumination-invariant façade recognition network (I2-FaçadeNet) that leverages sparsely gated multi-color space experts for enhanced façade image recognition in challenging illumination environments. First, RGB façade images are converted into multi-color spaces to eliminate the ambiguous texture in complex illumination. Second, we train expert networks using separate channels of multi-color spaces. Finally, a sparsely gated mechanism is introduced to manage the expert networks, enabling dynamic activation of expert networks and the merging of results. Experimental evaluations leveraging both the International Society for Photogrammetry and Remote Sensing benchmark data sets and the Shenzhen data sets reveal that our proposed I2 -FaçadeNet surpasses various depths of ResNet in façade recognition under complex illumination conditions. Specifically, the classification accuracy for poorly illuminated façades in Zurich improves by nearly 8%, while the accuracy for over-illuminated areas in Shenzhen increases by approximately 3%. Moreover, ablation studies conducted on façade images with complex illumination indicate that compared to traditional RGB-based ResNet, the proposed network achieves an accuracy improvement of 3% to 4% up to 100% for overexposed images and an accuracy improvement of 3% to 10% for underexposed images.

A Semi-Supervised Underexposed Image Enhancement Network With Supervised Context Attention and Multi-Exposure Fusion

Recently, image enhancement approaches yield impressive progress. However, most methods are still based supervised-learning, which requires plenty of paired data. Meanwhile, owing to the complex illumination condition in a real-world scenario, those methods trained on synthetic images cannot restore details in extremely dark or bright areas and lead to exposure errors. The traditional losses that deem all pixels the same in training also produce blurry edges in the result. To handle these problems, in this paper, we present an effective semi-supervised framework for severely underexposed image enhancement. Our network consists of a supervised and an unsupervised branch, which shares weights and can make full use of paired data and plenty of unpaired data. Meanwhile, a multi-exposure fusion module is designed to adaptively fuse the corrected images to address the low contrast and color bias issues occurring in some extreme situations. Moreover, we propose a supervised context attention module to better use the edge information as supervision to recover fine image details. Extensive experiments have proved that the proposed method outperforms state-of-the-art approaches in enhancing exposure images.

Estimation of chlorophyll content for urban trees from UAV hyperspectral images

Urban trees provide important ecosystem services to improve cities’ liveability and sustainability. Leaf chlorophyll content (Cab) estimation by remote sensing can help monitor tree health efficiently. However, the Cab retrieval of urban trees is challenging because of the complex canopy structures, backgrounds, and illuminations conditions. This paper proposed an automatic method for partitioning sunlit/shaded pixels and removal of bright-specular/dark-hole and background pixels. In addition, we proposed a new index, the Urban Tree Chlorophyll Index (UTCI), defined as UTCI=(ρ709-ρ697)/(ρ709-ρ686), based on the simulated hyperspectral images of urban tree using radiative transfer model. This proposed UTCI index outperforms existing narrow-band indices (NBIs) for estimating Cab for complex canopy structures, backgrounds, and illumination conditions evaluated using simulated hyperspectral images. The advantage of the UTCI was also demonstrated when applied to UAV hyperspectral images validated with direct field foliar measurements of Cab. It surpasses existing NBIs, demonstrating a moderate correlation (R2 = 0.34) with Cab under varying irradiance and a strong correlation (R2 = 0.62) with Cab under stable diffuse illumination. This study, for the first time, extensively investigated NBIs for Cab estimation of urban trees from UAV hyperspectral images, providing a theoretical and operational basis for future monitoring of Cab in the management of urban trees. This new method can be potentially applied to other vegetation types with complex canopy structures, backgrounds, and illuminations conditions.

Remote Sensing Image Semantic Segmentation Method Based on a Deep Convolutional Neural Network and Multiscale Feature Fusion

There are many problems with remote sensing images, such as large data scales, complex illumination conditions, occlusion, and dense targets. The existing semantic segmentation methods for remote sensing images are not accurate enough for small and irregular target segmentation results, and the edge extraction results are poor. The authors propose a remote sensing image segmentation method based on a DCNN and multiscale feature fusion. Firstly, an end-to-end remote sensing image segmentation model using complete residual connection and multiscale feature fusion was designed based on a deep convolutional encoder–decoder network. Secondly, weighted high-level features were obtained using an attention mechanism, which better preserved the edges, texture, and other information of remote sensing images. The experimental results on ISPRS Potsdam and Urban Drone datasets show that compared with the comparison methods, this method has better segmentation effect on small and irregular objects and achieves the best segmentation performance while ensuring the computation speed.

Target Localization Method Based on Image Degradation Suppression and Multi-Similarity Fusion in Low-Illumination Environments

Frame buildings as important nodes of urban space. The include high-speed railway stations, airports, residences, and office buildings, which carry various activities and functions. Due to illumination irrationality and mutual occlusion between complex objects, low illumination situations frequently develop in these architectural environments. In this case, the location information of the target is difficult to determine. At the same time, the change in the indoor electromagnetic environment also affects the location information of the target. Therefore, this paper adopts the vision method to achieve target localization in low-illumination environments by feature matching of images collected in the offline state. However, the acquired images have serious quality degradation problems in low-illumination conditions, such as low brightness, low contrast, color distortion, and noise interference. These problems mean that the local features in the collected images are missing, meaning that they fail to achieve a match with the offline database images; as a result, the location information of the target cannot be determined. Therefore, a Visual Localization with Multiple-Similarity Fusions (VLMSF) is proposed based on the Nonlinear Enhancement And Local Mean Filtering (NEALMF) preprocessing enhancement. The NEALMF method solves the problem of missing local features by improving the quality of the acquired images, thus improving the robustness of the visual positioning system. The VLMSF method solves the problem of low matching accuracy in similarity retrieval methods by effectively extracting and matching feature information. Experiments show that the average localization error of the VLMSF method is only 8 cm, which is 33.33% lower than that of the Kears-based VGG-16 similarity retrieval method. Meanwhile, the localization error is reduced by 75.76% compared with the Perceptual hash (Phash) retrieval method. The results show that the method proposed in this paper greatly alleviates the influence of low illumination on visual methods, thus helping city managers accurately grasp the location information of targets under complex illumination conditions.

AHF: An Automatic and Universal Image Preprocessing Algorithm for Circular-Coded Targets Identification in Close-Range Photogrammetry under Complex Illumination Conditions

In close-range photogrammetry, circular-coded targets (CCTs) are a reliable method to solve the issue of image correspondence. Currently, the identification methods for CCTs are very mature, but complex illumination conditions are still a key factor restricting identification. This article proposes an adaptive homomorphic filtering (AHF) algorithm to solve this issue, utilizing homomorphic filtering (HF) to eliminate the influence of uneven illumination. However, HF parameters vary with different lighting types. We use a genetic algorithm (GA) to carry out global optimization and take the identification result as the objective function to realize automatic parameter adjustment. This is different from the optimization strategy of traditional adaptive image enhancement methods, so the most significant advantage of the proposed algorithm lies in its automation and universality, i.e., users only need to input photos without considering the type of lighting conditions. As a preprocessing algorithm, we conducted experiments combining advanced commercial photogrammetric software and traditional identification methods, respectively. We cast stripe- and lattice-structured light to create complex lighting conditions, including uneven lighting, dense shadow areas, and elliptical light spots. Experiments showed that our algorithm significantly improves the robustness and accuracy of CCT identification methods under complex lighting conditions. Given the perfect performance under stripe-structured light, this algorithm can provide a new idea for the fusion of close-range photogrammetry and structured light. This algorithm helps to improve the quality and accuracy of photogrammetry and even helps to improve the decision making and planning process of photogrammetry.

Target Detection for Construction Machinery Based on Deep Learning and Multisource Data Fusion

Target detection in use-case environments is a challenging task, which is influenced by complex and dynamic landscapes, illumination, and vibrations. Therefore, this paper presents a research on road target detection based on deep learning by combining image data of vision with point cloud data from LiDAR. First, the depth map of the point cloud was densified for the sparse and disordered characteristics of the point cloud, with the ground removed to create an image dataset that could serve for training. Next, considering the computational capacity of the on-board processor and the accuracy, the MY3Net network is designed by integrating Mobilenet v2, a lightweight network, as the feature extractor, and YOLO v3, a high-precision network, as the multi-scale target detector, to implement the detection of RGB images and the densified depth maps. Finally, a decision-level fusion model is proposed to integrate the detection results of RGB images and depth maps with dynamic weights. Experimental results show that the proposed approach offers high detection accuracy even under complex illumination conditions.

Space Target Material Identification Based on Graph Convolutional Neural Network

Under complex illumination conditions, the spectral data distributions of a given material appear inconsistent in the hyperspectral images of the space target, making it difficult to achieve accurate material identification using only spectral features and local spatial features. Aiming at this problem, a material identification method based on an improved graph convolutional neural network is proposed. Superpixel segmentation is conducted on the hyperspectral images to build the multiscale joint topological graph of the space target global structure. Based on this, topological graphs containing the global spatial features and spectral features of each pixel are generated, and the pixel neighborhoods containing the local spatial features and spectral features are collected to form material identification datasets that include both of these. Then, the graph convolutional neural network (GCN) and the three-dimensional convolutional neural network (3-D CNN) are combined into one model using strategies of addition, element-wise multiplication, or concatenation, and the model is trained by the datasets to fuse and learn the three features. For the simulated data and the measured data, the overall accuracy of the proposed method can be kept at 85–90%, and their kappa coefficients remain around 0.8. This proves that the proposed method can improve the material identification performance under complex illumination conditions with high accuracy and strong robustness.

LCS-based Visual Recognition for AGV Guide Paths under Complex Illumination Conditions

Complex lighting is one of the most challenging problems in automatic guided vehicle (AGV) vision recognition system. In order to overcome the influence of uneven illumination on the accuracy and robustness of path recognition, this paper proposes the LCS based visual recognition for AGV guide paths under complex illumination conditions method, which converts the collected image into an invariant image through logarithmic chromaticity space (LCS) to eliminate the influence of illumination, and the minimum average entropy angle is used as the projection angle to generate the invariant image to improve the speed of image conversion to the invariant image. Experimental results show that the proposed method can effectively improve the robustness of AGV vision recognition system under complex lighting conditions.

Fault diagnosis of driving gear in rack and pinion drives based on multi-scale local binary pattern extraction and sparse representation

Rack and pinion drives (RPDs) are key components of battery-swapping systems (BSSs) used in electric heavy trucks; the faults occurring in these drives reduce the efficiency, accuracy, quality of battery swapping, and even pose potential safety risks. The operating conditions of RPD driving gear in BSSs are characterized by speed fluctuations, relatively low speeds, and reciprocating motion. To assess the driving gear fault characteristics under these conditions, based on the solution of image recognition under complex and low illumination conditions, this study proposes a fault diagnosis framework that includes adaptive down-sampling, three-dimensional acceleration data fusion, multi-scale local binary pattern (MS-LBP) extraction, and sparse representation. First, adaptive down-sampling is used to smooth out the speed fluctuation. Subsequently, MS-LBP extraction is employed to obtain efficient fault features at low speed. Finally, dictionary learning and sparse representations are conducted on the MS-LBP features. The effectiveness and advantages of the proposed diagnosis approach are demonstrated using monitoring data acquired from a BSS. Moreover, comparative studies demonstrate that the proposed fault diagnosis method yields improved performance.

A Line Segment Detector for Space Target Images Robust to Complex Illumination

The relative pose estimation of the space target is indispensable for on-orbit autonomous service missions. Line segment detection is an important step in pose estimation. The traditional line segment detectors show impressive performance under sufficient illumination, while it is easy to fail under complex illumination conditions where the illumination is too bright or too dark. We propose a robust line segment detector for space applications considering the complex illumination in space environments. An improved two-dimensional histogram construction strategy is used to optimize the Otsu method to improve the accuracy of anchor map extraction. To further improve line segment detection’s effect, we introduce an aggregation method that uses the angle difference between segments, the distance between endpoints, and the overlap degree of segments to filter the aggregation candidate segments and connect disjoint line segments that probably came from the same segment. We demonstrate the performance of the proposed line segment detector using a variety of images collected on a semiphysical simulation platform. The results show that our method has better performance than traditional line segment detectors including LSD, Linelet, etc., in terms of line detection precision.

An adaptive image enhancement approach for safety monitoring robot under insufficient illumination condition

The safety monitoring robots have been a potential solution to the task of patrol and inspection under uncontrolled complex environments such as tunnels. However, unclear semantic information and noises in low-light images will cause disturbance to the robot vision system. The deep learning-based enhancement is proven to advance the vision under poor illuminating conditions. However, the difficulties of excessive manual cost in collecting image pairs and image noise amplification are still challenging. In this work, we introduce a new adaptive image enhancement pipeline, which enables robots to self-adaptive to complex illumination conditions. With the pipeline, the IE-Module is proposed to learn image enhancement information from multi-scale feature blocks, and the SEnhance Module is used to eliminate the damage of denoising to normal pixels of the image. To make the algorithm better coupled with the safety monitoring robot, the algorithm is further enhanced with a filter threshold for adaptive to varied illumination conditions. The effectiveness of our method on both standard datasets and real-world scenarios was verified with experiments and real applications. The experiments on a low-light image benchmark as well as SLAM and 6D pose estimation experiments demonstrated that our method is superior to state-of-the-art methods qualitatively and quantitatively. Moreover, the proposed algorithm has been deployed in a real safety monitoring robot, which successfully demonstrates its capacity to perform patrol and inspection tasks in a low-light environment.

A Traffic Sign Recognition Method Under Complex Illumination Conditions

Environment awareness technology is one of the most critical technologies for autonomous vehicles, and traffic sign recognition is an important branch of environment awareness system, which has important research significance for ensuring traffic safety. To solve the problems of easy omission and inaccurate positioning for traffic sign detection under complex illumination conditions, there are two main innovations in this paper: firstly, an adaptive image enhancement algorithm was proposed to improve the image quality under complex illumination conditions; Secondly, a novel and lightweight attention block named Feature difference (FD) model was proposed to detect and recognize traffic signs. Unlike state-of-the-art attention models, which utilizes the difference between two feature maps to generate the attention mask. In this work, the single-stage target detection algorithm SSD was selected as the basic network, the backbone network was set as ResNet and VGG respectively and FD module was added. A large number of experiments were carried out to evaluate the optimization effect of FD module. The following conclusions can be drawn: image enhancement algorithms can provide better image samples; The FD model can be integrated into most of CNNs just by adding some shortcut connections, which does not introduce any additional parameters and layers; The average detection accuracy with FD module is 1.80% higher and the average recall rate was 1.51% higher than that without FD module, which has no great influence on the running speed. The FD module has better enhancement effect for dark image detection; The detection accuracy of the FD module is slightly better than other attention modules including BAM, CBAM and SE; The FD model can be used to evaluate the effect of the convolutional operation on features, which could help to find convolutional layers that have little effect on the accuracy of recognition for the network pruning.

Generating 3D Multispectral Point Clouds of Plants with Fusion of Snapshot Spectral and RGB-D Images.

Accurate and high-throughput plant phenotyping is important for accelerating crop breeding. Spectral imaging that can acquire both spectral and spatial information of plants related to structural, biochemical, and physiological traits becomes one of the popular phenotyping techniques. However, close-range spectral imaging of plants could be highly affected by the complex plant structure and illumination conditions, which becomes one of the main challenges for close-range plant phenotyping. In this study, we proposed a new method for generating high-quality plant 3-dimensional multispectral point clouds. Speeded-Up Robust Features and Demons was used for fusing depth and snapshot spectral images acquired at close range. A reflectance correction method for plant spectral images based on hemisphere references combined with artificial neural network was developed for eliminating the illumination effects. The proposed Speeded-Up Robust Features and Demons achieved an average structural similarity index measure of 0.931, outperforming the classic approaches with an average structural similarity index measure of 0.889 in RGB and snapshot spectral image registration. The distribution of digital number values of the references at different positions and orientations was simulated using artificial neural network with the determination coefficient (R 2) of 0.962 and root mean squared error of 0.036. Compared with the ground truth measured by ASD spectrometer, the average root mean squared error of the reflectance spectra before and after reflectance correction at different leaf positions decreased by 78.0%. For the same leaf position, the average Euclidean distances between the multiview reflectance spectra decreased by 60.7%. Our results indicate that the proposed method achieves a good performance in generating plant 3-dimensional multispectral point clouds, which is promising for close-range plant phenotyping.

Self-Supervised Leaf Segmentation under Complex Lighting Conditions

As an essential prerequisite task in image-based plant phenotyping, leaf segmentation has garnered increasing attention in recent years. While self-supervised learning is emerging as an effective alternative to various computer vision tasks, its adaptation for image-based plant phenotyping remains rather unexplored. In this work, we present a self-supervised leaf segmentation framework consisting of a self-supervised semantic segmentation model, a color-based leaf segmentation algorithm, and a self-supervised color correction model. The self-supervised semantic segmentation model groups the semantically similar pixels by iteratively referring to the self-contained information, allowing the pixels of the same semantic object to be jointly considered by the color-based leaf segmentation algorithm for identifying the leaf regions. Additionally, we propose to use a self-supervised color correction model for images taken under complex illumination conditions. Experimental results on datasets of different plant species demonstrate the potential of the proposed self-supervised framework in achieving effective and generalizable leaf segmentation.

Face Recognition Attendance System Method Based on Fusion of LBP and HOG

As one of the hot topics in the field of computer vision research, face recognition technology has received significant attention due to its potentiality for a wide range of applications in government as well as commercial purposes. In practical applications, although several existing face recognition methods have achieved good performances in specific scenes, they easily suffer from a sharp decline in recognition rate if affected by different conditions of light, expression, posture and occlusion. Among many factors, influences of complex illuminations on face recognition are particularly significant. To further improve the performance of the existing local binary pattern (LBP) operator, neighbourhood weighted average LBP (NWALBP) is first proposed for fully considering the strong correlations between pixel pairs in the neighbourhood, which extends the traditional LBP uni-layer neighbourhood template window to the bi-layer neighbourhood template window and calculates the weighted average of bi-layer neighbourhood pixels in each direction. Then, inspired by centre symmetric LBP (CS-LBP), centre symmetric NWALBP (CS-NWALBP) is further proposed, which can effectively reduce computation complexity by only comparing the weighted average values of the neighbourhood pixels that are symmetric about the centre pixel. Finally, by combining the merit of histogram of oriented gradient (HOG), a feature fusion algorithm named CS-NWALBP+HOG is suggested. Several experiments have eventually demonstrated that our proposed algorithms have more robust performance under complex illumination conditions if compared with many other latest algorithms.

Automatic Detection of Cracks on Concrete Surfaces in the Presence of Shadows.

Deep learning-based methods, especially convolutional neural networks, have been developed to automatically process the images of concrete surfaces for crack identification tasks. Although deep learning-based methods claim very high accuracy, they often ignore the complexity of the image collection process. Real-world images are often impacted by complex illumination conditions, shadows, the randomness of crack shapes and sizes, blemishes, and concrete spall. Published literature and available shadow databases are oriented towards images taken in laboratory conditions. In this paper, we explore the complexity of image classification for concrete crack detection in the presence of demanding illumination conditions. Challenges associated with the application of deep learning-based methods for detecting concrete cracks in the presence of shadows are elaborated on in this paper. Novel shadow augmentation techniques are developed to increase the accuracy of automatic detection of concrete cracks.

Research on Improved Gamma Transform Face Image Preprocessing Fusion Algorithm Under Complex Lighting Conditions

Introduction: Under complex illumination conditions, such as poor light sources and light changes rapidly, there are two disadvantages of current gamma transform in pre-processing face image: one is that the parameters of transformation need to be set based on experience; the other is that the details of the transformed image are not obvious enough. Objective: To improve the current gamma transform. Methods: This study proposes a weighted fusion algorithm of adaptive gamma transform and edge feature extraction. First, this paper proposes an adaptive gamma transform algorithm for face image pre-processing, that is, the parameter of transformation generated by calculation according to the specific gray value of the input face image. Secondly, this paper uses a Sobel edge detection operator to extract the edge information of the transformed image to get the edge detection image. Finally, this paper uses the adaptively transformed image and the edge detection image to obtain the final processing results through a weighted fusion algorithm. Results: The contrast of the face image after pre-processing is appropriate, and the details of the image are obvious. Conclusion: The method proposed in this paper can enhance the face image while retaining more face details, without human-computer interaction, and has a lower computational complexity degree.