Complex Lighting Environment Research Articles

The Adversarial Robust and Generalizable Stereo Matching for Infrared Binocular Based on Deep Learning

Despite the considerable success of deep learning methods in stereo matching for binocular images, the generalizability and robustness of these algorithms, particularly under challenging conditions such as occlusions or degraded infrared textures, remain uncertain. This paper presents a novel deep-learning-based depth optimization method that obviates the need for large infrared image datasets and adapts seamlessly to any specific infrared camera. Moreover, this adaptability extends to standard binocular images, allowing the method to work effectively on both infrared and visible light stereo images. We further investigate the role of infrared textures in a deep learning framework, demonstrating their continued utility for stereo matching even in complex lighting environments. To compute the matching cost volume, we apply the multi-scale census transform to the input stereo images. A stacked sand leak subnetwork is subsequently employed to address the matching task. Our approach substantially improves adversarial robustness while maintaining accuracy on comparison with state-of-the-art methods which decrease nearly a half in EPE for quantitative results on widely used autonomous driving datasets. Furthermore, the proposed method exhibits superior generalization capabilities, transitioning from simulated datasets to real-world datasets without the need for fine-tuning.

Semantic Segmentation Model-Based Boundary Line Recognition Method for Wheat Harvesting

The wheat harvesting boundary line is vital reference information for the path tracking of an autonomously driving combine harvester. However, unfavorable factors, such as a complex light environment, tree shade, weeds, and wheat stubble color interference in the field, make it challenging to identify the wheat harvest boundary line accurately and quickly. Therefore, this paper proposes a harvest boundary line recognition model for wheat harvesting based on the MV3_DeepLabV3+ network framework, which can quickly and accurately complete the identification in complex environments. The model uses the lightweight MobileNetV3_Large as the backbone network and the LeakyReLU activation function to avoid the neural death problem. Depth-separable convolution is introduced into Atrous Spatial Pyramid Pooling (ASPP) to reduce the complexity of network parameters. The cubic B-spline curve-fitting method extracts the wheat harvesting boundary line. A prototype harvester for wheat harvesting boundary recognition was built, and field tests were conducted. The test results show that the wheat harvest boundary line recognition model proposed in this paper achieves a segmentation accuracy of 98.04% for unharvested wheat regions in complex environments, with an IoU of 95.02%. When the combine harvester travels at 0~1.5 m/s, the normal speed for operation, the average processing time and pixel error for a single image are 0.15 s and 7.3 pixels, respectively. This method could achieve high recognition accuracy and fast recognition speed. This paper provides a practical reference for the autonomous harvesting operation of a combine harvester.

Light‐adaptive Mimicking Retina with In Situ Image Memorization via Resistive Switching Photomemristor Arrays

AbstractArtificial visual memory systems have been of particular interest since the development of machine vision and bionic robots. Ordinarily, the conventional system architecture requires the complex integration of two functional modules, a photo‐sensor and a memory device, which greatly limits the operating efficiency and increases the extra energy consumption. Nonetheless, other simply configured optoelectronics memory devices generally face challenges of adaption in complex light environments. Here, a resistive switching (RS) perovskite‐based photomemristor is presented that mimics the retina function. The dual function of light perception and in situ storage are both achieved. In the dark condition, it exhibits impressive memory performance with a high ON/OFF ratio of 104, a long retention time of over 104 s, and a low operating voltage of 0.38 V. With illumination, it shows self‐powered, broadband photo‐detecting characteristics with responsivity of 70 mA W−1 and detectivity of 7.5 × 1010 Jones. More importantly, benefiting from the material dual‐phase configuration, the highly steady photo‐adjusted RS windows are achieved. Its light‐adaptive memory application in dynamic environments is further demonstrated using the mimicking retina for a machine eye. This work can provide a strategy for enhanced RS photomemristor and its application in changing and varied scenarios.

Automatic Recognition and Quantification Feeding Behaviors of Nursery Pigs Using Improved YOLOV5 and Feeding Functional Area Proposals.

A novel method is proposed based on the improved YOLOV5 and feeding functional area proposals to identify the feeding behaviors of nursery piglets in a complex light and different posture environment. The method consists of three steps: first, the corner coordinates of the feeding functional area were set up by using the shape characteristics of the trough proposals and the ratio of the corner point to the image width and height to separate the irregular feeding area; second, a transformer module model was introduced based on YOLOV5 for highly accurate head detection; and third, the feeding behavior was recognized and counted by calculating the proportion of the head in the located feeding area. The pig head dataset was constructed, including 5040 training sets with 54,670 piglet head boxes, and 1200 test sets, and 25,330 piglet head boxes. The improved model achieves a 5.8% increase in the mAP and a 4.7% increase in the F1 score compared with the YOLOV5s model. The model is also applied to analyze the feeding pattern of group-housed nursery pigs in 24 h continuous monitoring and finds that nursing pigs have different feeding rhythms for the day and night, with peak feeding periods at 7:00-9:00 and 15:00-17:00 and decreased feeding periods at 12:00-14:00 and 0:00-6:00. The model provides a solution for identifying and quantifying pig feeding behaviors and offers a data basis for adjusting the farm feeding scheme.

Measurement of Vertical Dispersion and Pollution Impact of Artificial Light at Night in Urban Environment

Excessive artificial light disrupts our lives and poses health risks. In mixed-use urban areas, commercial lighting conflicts with residential activities. Vertical towers exacerbate the issue, as lower-level lighting affects higher-level occupants, and more research is needed to understand the vertical dispersion of light and its impact on sleep disturbance. This work offers a simple mathematical model for height dependence of illuminance in an urban building. It correlates basic measurements to reduce the values of the Exponent (n) and Linear (Coeff2) coefficients in a standard power-law curve. The collection of only a few data points can help to investigate the extent of the lighting spread at various floors of the building. Even with the complex lighting environment in Hong Kong, the methodology is exercised in 8 sample buildings from 4 different neighbourhoods to show the insights from the testing results. The heights of the studied buildings range from 3 to 18 stories, and they are for residential and mixed use. While the average measured illuminance at the street level was between 37 lx and 303 lx for the 4 neighbourhoods, the average illuminance across building heights was 14 lx to 247 lx. The existence of multiple light sources on building facades disrupts the simple inverse-square relationship with a single light source. The power n values range from approximately -1.7 to -0.8, while the Coeff2 values range from 61 to 7603 for the measured buildings. These variations indicate deviations from the assumption of a single source and suggest higher luminance and the presence of light sources at higher heights. With the simple inverse-square relationship, the proposed methodology can help to determine the height where the recommended limit can be met given the light situation at the street level.

All-weather real-time ore detection method based on near-infrared structured light and zero-crossing points characteristics

To address the low efficiency of ore detection methods caused by the complex lighting environment of mining sites, an all-weather real-time ore detection method based on near-infrared structured light and zero-crossing points characteristics was proposed. Firstly, an image acquisition method based on the infrared–visible radiation detection model (IVRDM) is proposed to avoid structured light imaging being disturbed by a strong sunlight environment; Then, the centerline of the light bar is extracted quickly and accurately based on the zero-crossing point characteristics; Finally, the light bar neighborhood splicing detection method was proposed to realize the real-time detection and 3D measurement of the target ore. The experimental results show that the proposed method can effectively resist strong sunlight interference and faster detection speed, the average extraction time of light bar centerline per frame is below 1.8 ms, and the error rate of ore size detection is <5 %.

Autonomous Control Method for Object Grasping of Logistics Sorting Manipulator Considering Changes in Lighting Environment

The autonomous grasping operation is the key to the intelligent logistics sorting manipulators. Since the current logistics sorting manipulators mostly use visual sensors to identify objects, they are extremely vulnerable to the changes in the lighting environment. Therefore, the research considers the influence of complex lighting environment on the autonomous grasping of logistics sorting manipulators. Furthermore, it verifies the effectiveness of SAC-AE-ICM algorithm through simulations and experiments. The experimental results show that SAC-AE-ICM algorithm can quickly achieve convergence in many experiments and can obtain global optimization. In the process of automatic capture, the success rate of SAC-AE-ICM algorithm can reach 90%, which is 25% higher than that of the method without ICM and has better convergence. The success rate was as high as 77% for unknown or irregular targets. In practical experiments, SAC-AE-ICM can effectively capture under good lighting conditions. However, under low light conditions, the probability of capturing unknown targets is about 72.5%. Overall, the success rates for single-object and multi-object capture are 88% and 85%, respectively.

Cross-directional consistency network with adaptive layer normalization for multi-spectral vehicle re-identification and a high-quality benchmark

To tackle the challenge of vehicle re-identification (Re-ID) in complex lighting environments and diverse scenes, multi-spectral sources like visible and infrared information are taken into consideration due to their excellent complementary advantages. However, multi-spectral vehicle Re-ID suffers cross-modality discrepancy caused by heterogeneous properties of different modalities as well as a big challenge of the diverse appearance with different views in each identity. Meanwhile, diverse environmental interference leads to heavy sample distributional discrepancy in each modality. In this work, we propose a novel cross-directional consistency network (CCNet) to simultaneously overcome the discrepancies from both modality and sample aspects. In particular, we design a new cross-directional center loss (Lcdc) to pull the modality centers of each identity close to mitigate cross-modality discrepancy, while the sample centers of each identity close to alleviate the sample discrepancy. Such a strategy can generate discriminative multi-spectral feature representations for vehicle Re-ID. In addition, we design an adaptive layer normalization unit (ALNU) to dynamically adjust individual feature distribution to handle distributional discrepancy of intra-modality features for robust learning. To provide a comprehensive evaluation platform, we create a high-quality RGB-NIR-TIR multi-spectral vehicle Re-ID benchmark (MSVR310), including 310 different vehicles from a broad range of viewpoints, time spans and environmental complexities. Comprehensive experiments on both created and public datasets demonstrate the effectiveness of the proposed approach comparing to the state-of-the-art methods. The dataset and code will be released for free academic usage at https://github.com/superlollipop123/Cross-directional-Center-Network-and-MSVR310.

P‐3.1: Research on Display Brightness Perception and Visual Comfort Representation Model

People's perception of brightness and visual comfort in different light environments are important indicators of display products. The perception of the brightness of the human eye is often different from the actual brightness of the display. This is because there is a difference between the perceived brightness and the physical brightness. The luminous body of the same physical brightness may give people different feelings, thus producing different perceived brightness. In the complex optical environment, the existing photoelectric measurement parameters may not be able to accurately describe the perception effect of the display device. The research on perceived brightness and visual comfort has a long history, but in view of the increasingly complex light environment and the development of diversified display equipment, the perceptual brightness model needs to be further modified and improved. Based on the modeling research of perceived brightness, this paper studies the perception mechanism of human eyes, analyzes the shortcomings of the current mainstream or new display devices, and proposes a more comfortable display technology based on people's viewing habits, ambient light and display content.

Radiometric Correction of Multispectral Field Images Captured under Changing Ambient Light Conditions and Applications in Crop Monitoring

Applications of unmanned aerial vehicle (UAV) spectral systems in precision agriculture require raw image data to be converted to reflectance to produce time-consistent, atmosphere-independent images. Complex light environments, such as those caused by varying weather conditions, affect the accuracy of reflectance conversion. An experiment was conducted here to compare the accuracy of several target radiance correction methods, namely pre-calibration reference panel (pre-CRP), downwelling light sensor (DLS), and a novel method, real-time reflectance calibration reference panel (real-time CRP), in monitoring crop reflectance under variable weather conditions. Real-time CRP used simultaneous acquisition of target and CRP images and immediate correction of each image. These methods were validated with manually collected maize indictors. The results showed that real-time CRP had more robust stability and accuracy than DLS and pre-CRP under various conditions. Validation with maize data showed that the correlation between aboveground biomass and vegetation indices had the least variation under different light conditions (correlation all around 0.74), whereas leaf area index (correlation from 0.89 in sunny conditions to 0.82 in cloudy days) and canopy chlorophyll content (correlation from 0.74 in sunny conditions to 0.67 in cloudy days) had higher variation. The values of vegetation indices TVI and EVI varied little, and the model slopes of NDVI, OSAVI, MSR, RVI, NDRE, and CI with manually measured maize indicators were essentially constant under different weather conditions. These results serve as a reference for the application of UAV remote sensing technology in precision agriculture and accurate acquisition of crop phenotype data.

A review of the effect of the light environment of the VDT workspace on the "learning to learn" effect of video game training.

In recent years, the role of video games in enhancing brain plasticity and learning ability has been verified, and this learning transfer is known as the "learning to learn" effect of video game training. At the same time, against the background of healthy lighting, the influence of non-visual effects of light environment on the human rhythmic system has been gradually confirmed. As a special operation form of Visual Display Terminal (VDT) operation, video game training has a high dependence on VDT equipment and the VDT screen, and the background usually has a huge difference in brightness. Compared with the light environment of ordinary operation space, the light environment of VDT operation space is more complex. This complex light environment's non-visual effects cause human emotions, alertness, fatigue, cognitive ability, and other changes, which may affect the efficiency of the "learning to learn" effect of video game training. This article focuses on the impact of the light environment in the VDT workspace on the "learning to learn" effect of video game training. It first traces the factors that trigger the "learning to learn" effect of video game training, that is, the improvement of people's attention, perception, and cognitive ability. Then, the influencing mechanism and the evaluation method of the VDT workspace space light environment on the human rhythm system are discussed based on the basic theory of photobiological effect. In addition, the VDT display lighting light time pattern, photophysical properties, regulation, and protection mechanism on the human rhythm system are studied to demonstrate the VDT workspace light environment's special characteristics. Finally, combined with the progress of artificial lighting technology and the research results of health lighting, given the "learning to learn" effect of video game training, some thoughts on the design of the light environment of the workplace and future research directions are presented.

Improving AR-SSVEP Recognition Accuracy Under High Ambient Brightness Through Iterative Learning.

Augmented reality-based brain-computer interface (AR-BCI) system is one of the important ways to promote BCI technology outside of the laboratory due to its portability and mobility, but its performance in real-world scenarios has not been fully studied. In the current study, we first investigated the effect of ambient brightness on AR-BCI performance. 5 different light intensities were set as experimental conditions to simulate typical brightness in real scenes, while the same steady-state visual evoked potentials (SSVEP) stimulus was displayed in the AR glass. The data analysis results showed that SSVEP can be evoked under all 5 light intensities, but the response intensity became weaker when the brightness increased. The recognition accuracies of AR-SSVEP were negatively correlated to light intensity, the highest accuracies were 89.35% with FBCCA and 83.33% with CCA under 0 lux light intensity, while they decreased to 62.53% and 49.24% under 1200 lux. To solve the accuracy loss problem in high ambient brightness, we further designed a SSVEP recognition algorithm with iterative learning capability, named ensemble online adaptive CCA (eOACCA). The main strategy is to provide initial filters for high-intensity data by iteratively learning low-light-intensity AR-SSVEP data. The experimental results showed that the eOACCA algorithm had significant advantages under higher light intensities ( 600 lux). Compared with FBCCA, the accuracy of eOACCA under 1200 lux was increased by 13.91%. In conclusion, the current study contributed to the in-depth understanding of the performance variations of AR-BCI under different lighting conditions, and was helpful in promoting the AR-BCI application in complex lighting environments.

Illumination-robust milling surface roughness machine vision inspection based on MAML++ network

Existing roughness machine vision inspection methods based on designed indices require the artificial construction of features associated with roughness based on image information. Such methods are less universal but highly demanding for the imaging environment and cannot be directly applied to general industrial environments. The roughness machine vision inspection methods based on traditional deep learning, on the other hand, rely heavily on a large number of training samples, and the model needs to be retrained when the recognition of new categories is required by the inspection task. Therefore, a machine vision inspection method for milling surface roughness grade based on the improved model-agnostic meta-learning (MAML++) network was proposed to address the above problems. This method could not only automatically extract the image features associated with roughness but also recognize the surface roughness grade of milling workpieces by a small number of labeled samples. It reduced the need for sample size and can recognize new categories without retraining the model. The experimental results showed that the method exhibited high accuracy in roughness inspection tasks with different sample distributions and had promising robustness to complex light environments. In addition, in terms of the requirement for recognizing new roughness categories, a low-cost and fast roughness inspection task could be accomplished by acquiring fewer labeled samples. In summary, the proposed model has the potential to be a fast inspection system that can be applied to surface roughness inspection for most industrial applications. It provides the technology for inline inspection of machine vision roughness in general industrial lighting environments.

Photosynthetic mechanism of maize yield under fluctuating light environments in the field.

The photosynthetic mechanism of crop yields in fluctuating light environments in the field remains controversial. To further elucidate this mechanism, we conducted field and simulation experiments using maize (Zea mays) plants. Increased planting density enhanced the light fluctuation frequency and reduced the duration of daily high light, as well as the light-saturated photosynthetic rate, biomass, and yield per plant. Further analysis confirmed a highly significant positive correlation between biomass and yield per plant and the duration of photosynthesis related to daily high light. The simulation experiment indicated that the light-saturated photosynthetic rate of maize leaves decreased gradually and considerably when shortening the daily duration of high light. Under an identical duration of high light exposure, increasing the fluctuation frequency decreased the light-saturated photosynthetic rate slightly. Proteomic data also demonstrated that photosynthesis was mainly affected by the duration of high light and not by the light fluctuation frequency. Consequently, the current study proposes that an appropriate duration of daily high light under fluctuating light environments is the key factor for greatly improving photosynthesis. This is a promising mechanism by which the photosynthetic productivity and yield of maize can be enhanced under complex light environments in the field.

Research on the Visual Servoing method based on semantic segmentation results of metal objects in a complex lighting environment

The problem of feature recognition and servo tracking for semantic segmentation results of metal parts under complex and variable lighting conditions is addressed. In this paper, we propose an algorithm that incorporates template matching and image-based visual servoing based on semantic segmentation. The method adopts a geometry-based template matching method to stably identify the image features of the pixel region obtained from semantic segmentation, introduces a feature pyramid dimensionality reduction method to improve the processing speed, and then introduces the extracted image moment features into the image-based visual servoing to operate the robotic arm to perform the servo process on the target.

Recognizing Teachers’ Hand Gestures for Effective Non-Verbal Interaction

Hand gesturing is one of the most useful non-verbal behaviors in the classroom, and can help students activate multi-sensory channels to complement teachers’ verbal behaviors and ultimately enhance teaching effectiveness. The existing mainstream detection algorithms that can be used to recognize hand gestures suffered from low recognition accuracy under complex backgrounds and different backlight conditions. This study proposes an improved hand gesture recognition framework based on key point statistical transformation features. The proposed framework can effectively reduce the sensitivity of images to background and light conditions. We extracted key points of the image and establish a weak classifier to enhance the anti-interference ability of the algorithm in the case of noise and partial occlusion. Then, we used a deep convolutional neural network model with multi-scale feature fusion to recognize teachers’ hand gestures. A series of experiments were conducted on different human gesture datasets to verify the performance of the proposed framework. The results show that the framework proposed in this study has better detection and recognition rates compared to the you only look once (YOLO) algorithm, YOLOv3, and other counterpart algorithms. The proposed framework not only achieved 98.43%, measured by F1 score, for human gesture images in low-light conditions, but also has good robustness in complex lighting environments. We used the proposed framework to recognize teacher gestures in a case classroom setting, and found that the proposed framework outperformed YOLO and YOLOv3 algorithms on small gesture images with respect to recognition performance and robustness.

A Neural Network Structure with Attention Mechanism and Additional Feature Fusion Layer for Tomato Flowering Phase Detection in Pollination Robots

In recent years, convolutional neural networks have made many advances in the field of computer vision. In smart greenhouses, using robots based on computer vision technology to pollinate flowers is one of the main methods of pollination. However, due to the complex lighting environment and the influence of leaf shadow in the greenhouse, it is difficult for the existing object detection algorithms to have high recall rate and accuracy. Based on this problem, from the perspective of application, we proposed a Yolov5s-based tomato flowering stage detection method named FlowerYolov5, which can well identify the bud phase, blooming phase and first fruit phase of tomato flowers. Firstly, in order to reduce the loss of tomato flower feature information in convolution and to strengthen the feature extraction of the target, FlowerYolov5 adds a new feature fusion layer. Then, in order to highlight the information of the object, the Convolutional Block Attention module (CBAM) is added to the backbone layer of FlowerYolov5. In the constructed tomato flower dataset, compared with YOLOv5s, the mAP of FlowerYolov5 increased by 7.8% (94.2%), and the F1 score of FlowerYolov5 increased by 6.6% (89.9%). It was found that the overall parameter of FlowerYolov5 was 23.9 Mbyte, thus achieving a good balance between model parameter size and recognition accuracy. The experimental results show that the FlowerYolov5 has good robustness and more accurate precision. At the same time, the recall rate has also been greatly improved. The prediction results of the proposed algorithm can provide more accurate flower positioning for the pollination robot and improve its economic benefits.

Determining scalar illuminance from cubic illuminance data. Part 2: Tests in real lighting environments and an approach to improve its accuracy

Scalar illuminance, which describes the constant illumination from all directions, is an important indicator of the abundance of light for a lit object and the adequacy of illumination perceived. This paper proposes a more reliable method to recover scalar illuminance based on tests in natural complex lighting environments. The performance of Cuttle’s Approach 1, Mangkuto’s Approach 2 and Approach 3, together with Xia et al.’s potential Approach 4, were tested under a total of 610 high dynamic range (HDR) panoramic maps of real scenes. The relationships between predicted scalar illuminance and normalised diffuseness levels were checked. The results indicate that the potential Approach 4 is more robust to the cubic meter’s postures, and the predicted scalar illuminance has a regular relationship with normalised diffuseness levels. Approach 4 was corrected, together with Approach 1, formulating a new method named Approach 5S. Later, the proposed Approach 5S was evaluated under 205 indoor and 2233 outdoor panoramas from the Laval HDR databases, and it was shown to recover more reliable scalar illuminance with an average error within 5% in general. This study has provided a practical solution to more accurate vector illuminance-based metrics in real lighting environments. This algorithm can be further integrated into the development of cubic illumination meter instruments.

Research on Inversion and Correction Method of Urban Light Environment Based on Cooperative Observation

With the continuously growing city size and the increasingly complex and changeable light environment in the city, remote sensing and ground-measured technologies have certain limitations in the research of urban night light environment. The ground-measured data are accurate but low in efficiency and small in scale, while the night-light remote sensing data have the characteristics of high accuracy and large coverage. In this paper, high-resolution night-light remote sensing data and high-accuracy ground-measured data were used to establish an urban ground light environment inversion method with the advantages of remote sensing and ground-measured data in a “space-ground collaboration” approach. A ground database is constructed in GIS based on 26,000 ground measurement data of 4 blocks, 3 spatial perspectives, and 3 light environment parameters. Based on the comparison of the numerical relationship between the measured data of each light environment parameter and each window, the horizontal window is selected as the target window for the ground night light environment inversion research. The urban night light environment inversion method based on the correlation between telemetry and ground- measurement is used to construct and compare the correlation between Luojia night light radiance data and 9 sets of measured data of different ground windows and different light environment parameters. The illuminance measured data of horizontal window and Luojia radiance data, both of which are highly correlated, are selected for regression analysis. The mathematical inversion model of ground illuminance is constructed based on the cubic polynomial model with the lowest RMSE among the six regression models. The inversion result not only has photometric calibration, but also is superior to the original data in terms of population data relevance and accuracy.

Face detection from patterns of shading and shadows: The role of overhead illumination in generating the familiar appearance of the human face

Face detection in human vision relies on a stereotypical pattern of visual features common to different faces. How are these visual features generated in the environment? Here we investigate how characteristic patterns of shading and shadows that occur across the face act as a cue for face detection. We use 3D rendering to isolate facial shading under simulated lighting conditions, comparing the broad patterns of contrast that occur across the face when light arrives from different angles. We find that human performance in discriminating faces from non-face objects using these contrast patterns depends strongly on the lighting direction. In particular, light arriving from above the brow tends to facilitate face detection – consistent with the statistics of real-world lighting environments, in which light commonly arrives more strongly from above. Indeed, in a further experiment, we find that asymmetries in lighting that occur in complex and naturalistic lighting environments produce contrast patterns across the face that facilitate face detection. These effects occurred independent of the lighting direction relative to the viewer, suggesting that cues to face detection emerge from the interaction between face morphology and vertical asymmetries in lighting direction, independent of the viewer's knowledge or expectations about lighting direction. Comparison with the performance of an image classifier suggests that the effects of lighting direction partly reflect differences in image information that result from the interaction between shape and illumination, as well as face detection in human observers being better-tuned to the pattern of shading and shadows that occurs across an upright face that is lit from overhead.