Illumination Variations Research Articles

Deep Multimodal Fusion of Visual and Auditory Features for Robust Material Recognition

This paper presents a deep neural network incorporating visual and auditory data fusion to enhance material recognition performance. Traditional recognition techniques relying on single data modalities face accuracy and robustness limitations, especially in complex real-world environments. To address these challenges, we develop a multimodal fusion-based model. The proposed approach first extracts features from input images and sounds separately using CNNs and spectral analysis. A concatenation layer then integrates the visual and auditory features. Extensive experiments demonstrate superior material classification over uni-modal methods, with 100% test accuracy across seven material types. The multi-modal fusion model also demonstrates stronger resilience to noise and illumination variations. This research provides a valuable foundation for robust material perception in intelligent systems.

Microprocessor Control of the Electric Drive of Variable Radiation Installation and Ensuring of Operation Reliability

Introduction. The introduction of energy-saving technologies has become particularly relevant in recent years, when energy prices have been constantly increasing. This is particularly important for greenhouse farms, where the main condition for growing plants indoors is the creation of artificial climatic conditions: temperature, humidity, and lighting.Problem Statement. Given the light and the dark stages of photosynthesis, it becomes possible to use a variable light field in the plant-growing photoculture, which is created by light installations with variable radiation. The development of an energy-saving electric drive and automated control systems for the parameters of artificial climatic conditions when growing plants indoors results in reducing energy consumption and increasing the reliability of the technological process in greenhouses of agro-industrial complex.Purpose. Implementation of energy-saving variable illumination irradiating systems with an automated control system for greenhouse farms.Materials and Methods. A lighting installation with rotation of the irradiator in the vertical and horizontal planes with a discrete electric drive and microprocessor control has been used.Results. The current state of variable irradiating systems with energy-saving radiation sources has been analyzed. The use of a discrete electric drive with an automated control system has been proposed. The installation with a DNAT-400 radiation source and a ZHO-01 reflector enables irradiating an area of up to 100 m2. The maximum exposure to photosynthetically active radiation is 30 W/m2, and the irradiation irregularity does not exceed 20%. At the same time, electricity savings reach about 10 thousand kWh per season. The yield and quality of products remain unchanged.Conclusions. The results of the research have confirmed the feasibility of practical implementation of variable irradiation system for energy efficiency of greenhouse operations.

A review on video person re-identification based on deep learning

Person Re-Identification (ReID) is an essential technology for matching a person across non-overlapping cameras. It has attracted increasing attention in recent years due to its wide range of applications in various real-world scenarios such as security surveillance and criminal investigation. Different from other person ReID tasks, video-based ReID uses a video clip as the retrieval input, which can provide more promising ReID performance because that the video has rich information on appearance, motion cues and pose variations on temporal pipeline. Over the last few years, many deep learning-based video person ReID have been proposed to address various challenges, such as illumination variation, complex background, occlusion, etc. To provide a more comprehensive and readable review on existing video-based person ReID methods, we propose a novel taxonomy method that observes existing methods from four perspectives: data, algorithms, computing power, and applications. Specifically, we first introduce some popular datasets and evaluation criterion used for video-based person ReID. Next, from limited data and little annotation view, we introduce data augmentation and unsupervised learning ReID. From algorithm view, we focus on reviewing supervised methods including spatial feature learning, temporal feature learning and spatio-temporal feature learning, and further discuss and conduct a systematic comparison among these approaches. From complex open-world application view, we mainly summarized domain adaption and multimodal ReID. From insufficient GPU computing power view, we mainly discuss modality-agnostic unified large-scale ReID and their lightweighting. Finally, we provide a discussion of open problems and potential research directions for the community.

An efficient method for water content estimation of building materials from spectral reflectance

We propose a nondestructive methodology to accurately estimate the water content of building materials from spectral reflectance in the shortwave infrared. The water content of a wet sample is estimated from the relative position of its reflectance spectrum on the curve between 2 reference spectra with known water content. By design, the approach is invariant to variations in illumination and acquisition conditions. Validation is done on datasets of clay powders and bricks. The experimental results provide confirmation of the effectiveness of the proposed method.

Improving Radiometric Block Adjustment for UAV Multispectral Imagery under Variable Illumination Conditions

Unmanned aerial vehicles (UAVs) equipped with multispectral cameras offer great potential for applications in precision agriculture. A critical challenge that limits the deployment of this technology is the varying ambient illumination caused by cloud movement. Rapidly changing solar irradiance primarily affects the radiometric calibration process, resulting in reflectance distortion and heterogeneity in the final generated orthomosaic. In this study, we optimized the radiometric block adjustment (RBA) method, which corrects for changing illumination by comparing adjacent images and from incidental observations of reference panels to produce accurate and uniform reflectance orthomosaics regardless of variable illumination. The radiometric accuracy and uniformity of the generated orthomosaic could be enhanced by improving the weights of the information from the reference panels and by reducing the number of tie points between adjacent images. Furthermore, especially for crop monitoring, we proposed the RBA-Plant method, which extracts tie points solely from vegetation areas, to further improve the accuracy and homogeneity of the orthomosaic for the vegetation areas. To validate the effectiveness of the optimization techniques and the proposed RBA-Plant method, visual and quantitative assessments were conducted on a UAV-image dataset collected under fluctuating solar irradiance conditions. The results demonstrated that the optimized RBA and RBA-Plant methods outperformed the current empirical line method (ELM) and sensor-corrected approaches, showing significant improvements in both radiometric accuracy and homogeneity. Specifically, the average root mean square error (RMSE) decreased from 0.084 acquired by the ELM to 0.047, and the average coefficient of variation (CV) decreased from 24% (ELM) to 10.6%. Furthermore, the orthomosaic generated by the RBA-Plant method achieved the lowest RMSE and CV values, 0.039 and 6.8%, respectively, indicating the highest accuracy and best uniformity. In summary, although UAVs typically incorporate lighting sensors for illumination correction, this research offers different methods for improving uniformity and obtaining more accurate reflectance values from orthomosaics.

UAV Visual Tracking with Enhanced Feature Information

Unmanned aerial vehicles (UAVs) visual tracking is an important research direction. The tracking object is lost due to the problems of target occlusion, illumination variation, flight vibration and so on. Therefore, based on a Siamese network, this study proposes a UAVs visual tracker named SiamDFT++ to enhance the correlation of depth features. First, the network width of the three-layer convolution after the full convolution neural network is doubled, and the appearance information of the target is fully utilized to complete the feature extraction of the template frame and the detection frame. Then, the attention information fusion module and feature deep convolution module are proposed in the template branch and the detection branch, respectively. The feature correlation calculation methods of the two depths can effectively suppress the background information, enhance the correlation between pixel pairs, and efficiently complete the tasks of classification and regression. Furthermore, this study makes full use of shallow features to enhance the extraction of object features. Finally, this study uses the methods of deep cross-correlation operation and complete intersection over union to complete the matching and location tasks. The experimental results show that the tracker has strong robustness in UAVs short-term tracking scenes and long-term tracking scenes.

PSDF: Prior-Driven Neural Implicit Surface Learning for Multi-view Reconstruction.

Surface reconstruction has traditionally relied on the Multi-View Stereo (MVS)-based pipeline, which often suffers from noisy and incomplete geometry. This is due to that although MVS has been proven to be an effective way to recover the geometry of the scenes, especially for locally detailed areas with rich textures, it struggles to deal with areas with low texture and large variations of illumination where the photometric consistency is unreliable. Recently, Neural Implicit Surface Reconstruction (NISR) combines surface rendering and volume rendering techniques and bypasses the MVS as an intermediate step, which has emerged as a promising alternative to overcome the limitations of traditional pipelines. While NISR has shown impressive results on simple scenes, it remains challenging to recover delicate geometry from uncontrolled real-world scenes which is caused by its underconstrained optimization. To this end, the framework PSDF is proposed which resorts to external geometric priors from a pretrained MVS network and internal geometric priors inherent in the NISR model to facilitate high-quality neural implicit surface learning. Specifically, the visibility-aware feature consistency loss and depth prior-assisted sampling based on external geometric priors are introduced. These proposals provide powerfully geometric consistency constraints and aid in locating surface intersection points, thereby significantly improving the accuracy and delicate reconstruction of NISR. Meanwhile, the internal prior-guided importance rendering is presented to enhance the fidelity of the reconstructed surface mesh by mitigating the biased rendering issue in NISR. Extensive experiments on Tanks and Temples datasets show that PSDF achieves state-of-the-art performance on complex uncontrolled scenes.

AI-ENHANCED TRACKSEGNET AN ADVANCED MACHINE LEARNING TECHNIQUE FOR VIDEO SEGMENTATION AND OBJECT TRACKING

Video segmentation and object tracking are critical tasks in computer vision with applications spanning surveillance, autonomous driving, and interactive media. Traditional methods often struggle with the dynamic nature of video data, where object occlusions, variations in illumination, and complex motion patterns present significant challenges. Existing segmentation and tracking systems frequently suffer from inaccuracies in handling real-time video sequences, particularly in distinguishing and tracking multiple overlapping objects. The limitations of current models in addressing these issues necessitate the development of more advanced techniques that can effectively manage dynamic scenes and improve tracking accuracy. To address these challenges, we propose an advanced machine learning technique, AI-Enhanced TrackSegNet, which integrates deep learning with novel attention mechanisms for improved video segmentation and object tracking. Our method utilizes a combination of Convolutional Neural Networks (CNNs) for feature extraction and Long Short-Term Memory (LSTM) networks for temporal sequence modeling. We introduce an attention-based mechanism to dynamically focus on relevant features, enhancing the model's ability to handle occlusions and varying object appearances. The model was trained on a diverse dataset of video sequences, incorporating both synthetic and real-world footage. The AI-Enhanced TrackSegNet demonstrated significant improvements in performance compared to existing techniques. Our method achieved an average Intersection over Union (IoU) score of 86.7% for segmentation and a tracking precision rate of 91.3% on the MOT17 benchmark dataset. These results represent a 10.2% improvement in IoU and a 7.5% increase in tracking precision compared to state-of-the-art methods. The model also exhibited enhanced robustness in complex scenes, handling occlusions and motion variations with greater accuracy.

A change detection algorithm for the SAR images based on DWT and DE optimization

ABSTRACT In this paper, a novel change detection algorithm is proposed for the remote sensing Synthetic Aperture Radar (SAR) images. Change detection in SAR images is one of the critical tasks in the field of remote sensing as the images contain significant speckle noise and illumination variations. In order to address these issues, a novel change detection algorithm is proposed using Discrete Wavelet Transform (DTW) and Differential Evaluation (DE). At first, the DWT is utilized to remove the speckle noise from the images. Then, a DE optimization-based Support Vector Machine (SVM) algorithm is proposed to detect the changes in SAR images. The proposed method can give comparatively better correct classification values, and lesser false alarm as well as total error values than the recently developed change detection methods.

Illusory light drives pupil responses in primates.

In humans, the eye pupils respond to both physical light sensed by the retina and mental representations of light produced by the brain. Notably, our pupils constrict when a visual stimulus is illusorily perceived brighter, even if retinal illumination is constant. However, it remains unclear whether such perceptual penetrability of pupil responses is an epiphenomenon unique to humans or whether it represents an adaptive mechanism shared with other animals to anticipate variations in retinal illumination between successive eye fixations. To address this issue, we measured the pupil responses of both humans and macaque monkeys exposed to three chromatic versions (cyan, magenta, and yellow) of the Asahi brightness illusion. We found that the stimuli illusorily perceived brighter or darker trigger differential pupil responses that are very similar in macaques and human participants. Additionally, we show that this phenomenon exhibits an analogous cyan bias in both primate species. Beyond evincing the macaque monkey as a relevant model to study the perceptual penetrability of pupil responses, our results suggest that this phenomenon is tuned to ecological conditions because the exposure to a "bright cyan-bluish sky" may be associated with increased risks of dazzle and retinal damages.

Application of Digital Tools and Artificial Intelligence to the Myasthenia Gravis Core Examination.

Advances in video image analysis and artificial intelligence provide the opportunity to transform the approach to patient evaluation through objective digital evaluation. We assessed ability to quantitate Zoom video recordings of a standardized neurological examination the myasthenia gravis core examination (MG-CE), which had been designed for telemedicine evaluations. We used Zoom (Zoom Video Communications) videos of patients with myasthenia gravis undergoing the MG-CE. Computer vision in combination with artificial intelligence methods were used to build algorithms to analyze videos with a focus on eye or body motions. For the assessment of examinations involving vocalization, signal processing methods were developed, including natural language processing. A series of algorithms were built that could automatically compute the metrics of the MG-CE. Fifty-one patients with MG with videos recorded twice on separate days and 15 control subjects were assessed once. We were successful in quantitating lid, eye, and arm positions and as well as well as develop respiratory metrics using breath counts. Cheek puff exercise was found to be of limited value for quantitation. Technical limitations included variations in illumination, bandwidth, and recording being done on the examiner side, not the patient. Several aspects of the MG-CE can be quantitated to produce continuous measures via standard Zoom video recordings. Further development of the technology offer the ability for trained, non-physician, health care providers to perform precise examination of patients with MG outside the clinic, including for clinical trials. Advances in video image analysis and artificial intelligence provide the opportunity to transform the approach to patient evaluation. Here, we asked whether video recordings of the typical telemedicine examination for the patient with myasthenia gravis be used to quantitate examination findings? Despite recordings not made for purpose, we were able to develop and apply computer vision and artificial intelligence to Zoom recorded videos to successfully quantitate eye muscle, facial muscle, and limb fatigue. The analysis also pointed out limitations of human assessments of bulbar and respiratory assessments. The neuromuscular examination can be enhanced by advance technologies, which have the promise to improve clinical trial outcome measures as well as standard care.

Universal Facial Encoding of Codec Avatars from VR Headsets

Faithful real-time facial animation is essential for avatar-mediated telepresence in Virtual Reality (VR). To emulate authentic communication, avatar animation needs to be efficient and accurate: able to capture both extreme and subtle expressions within a few milliseconds to sustain the rhythm of natural conversations. The oblique and incomplete views of the face, variability in the donning of headsets, and illumination variation due to the environment are some of the unique challenges in generalization to unseen faces. In this paper, we present a method that can animate a photorealistic avatar in realtime from head-mounted cameras (HMCs) on a consumer VR headset. We present a self-supervised learning approach, based on a cross-view reconstruction objective, that enables generalization to unseen users. We present a lightweight expression calibration mechanism that increases accuracy with minimal additional cost to run-time efficiency. We present an improved parameterization for precise ground-truth generation that provides robustness to environmental variation. The resulting system produces accurate facial animation for unseen users wearing VR headsets in realtime. We compare our approach to prior face-encoding methods demonstrating significant improvements in both quantitative metrics and qualitative results.

Accurate Vision-Based PCB Positioning Using Cosine-Convolutional Neural Networks

Object positioning is a fundamental task for assembly and inspection operations in the electronic industry. Traditional methods such as template matching and feature-point detection have been applied for object positioning. They are however computationally expensive and are generally affected by environmental changes. Deep learning models based on Convolutional Neural Network (CNN) have also been used for object positioning. They are computationally very efficient but can be sensible under environmental variations such as illumination and noise. In this article, a Cosine-Convolution operation is proposed with the objective of minimizing the effects of illumination variations. The proposed Cosine-Convolution can substitute any convolutional operation in a CNN-based regressor for positioning tasks. The proposed convolution is based on cosine-measure that normalizes the convolution between an image window and a filter at local scope. The proposed Cosine-Convolutional Neural Network (Cosine-CNN) performs more accurate than the traditional CNN under environment variations, including illumination changes, noise, defocusing and template occlusion. The experimental results reveal that the proposed model can reach mean prediction errors smaller than 1 pixel and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1^\circ$</tex-math> </inline-formula> for object shift and rotation, regardless of the environmental variations. The method is computationally very fast, with an average evaluation time of 1.2-ms for real-time PCB positioning. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —The training of deep learning models based on Convolutional Neural Network (CNN) can be tedious. Image samples with different illumination settings must be provided to have an efficient and robust model, which may not be easy to acquire or synthesize. This study proposes a solution to the challenge of illumination variations in images for the object positioning task in Printed Circuit Boards (PCB). A new Convolutional operation called Cosine-Convolutional that normalizes the traditional Convolution is introduced to overcome the illumination variations in the input images. Experimental results demonstrate that the proposed Cosine-Convolutional operation is less sensitive against lightning variations with respect to the training images.

Spatio-Temporal Attention and Gaussian Processes for Personalized Video Gaze Estimation.

Gaze is an essential prompt for analyzing human behavior and attention. Recently, there has been an increasing interest in determining gaze direction from facial videos. However, video gaze estimation faces significant challenges, such as understanding the dynamic evolution of gaze in video sequences, dealing with static backgrounds, and adapting to variations in illumination. To address these challenges, we propose a simple and novel deep learning model designed to estimate gaze from videos, incorporating a specialized attention module. Our method employs a spatial attention mechanism that tracks spatial dynamics within videos. This technique enables accurate gaze direction prediction through a temporal sequence model, adeptly transforming spatial observations into temporal insights, thereby significantly improving gaze estimation accuracy. Additionally, our approach integrates Gaussian processes to include individual-specific traits, facilitating the personalization of our model with just a few labeled samples. Experimental results confirm the efficacy of the proposed approach, demonstrating its success in both within-dataset and cross-dataset settings. Specifically, our proposed approach achieves state-of-the-art performance on the Gaze360 dataset, improving by 2.5° without personalization. Further, by personalizing the model with just three samples, we achieved an additional improvement of 0.8°. The code and pre-trained models are available at https://github.com/jswati31/stage.

Evaluating Learning-based Tie Point Matching for Geometric Processing of Off-Track Satellite Stereo

Abstract. Tie-point matching of off-track stereo images is a very challenging task, which can impact bias compensation and digital surface model (DSM) generation. Compared to in-track stereo images, off-track stereo images are more complex primarily due to the radiometric differences caused by sun illumination, sensor responses, atmospheric conditions, and seasonal land cover variations, and secondly due to the longer baseline and larger intersection angle. These challenges significantly limit the use of the vast number of images in satellite archives for automated geometric processing and mapping. Recent advances in deep learning (DL) based matching show promising results against images with diverse illuminations, viewing angles and scales through learning examples. This paper evaluates the potentials of addressing the tie point matching problems in off-track satellite stereo images. Specifically, we focus on stereo pairs that failed or underperformed in classic matching algorithms (i.e., SIFT (scale invariant feature transform)), and evaluate the DL-based tie points matchers by its resulting geometric accuracy in relative orientation, and the generated DSM. The experiments are carried out using 40 off-track satellite stereo pairs from four different regions around the world. We conclude that DL-based methods provide a significant higher success rate in matching challenging multi-temporal stereo pairs, even if their matching accuracy is slightly lower than classic algorithms.

Spatial-temporal graph Transformer for object tracking against noise spoofing interference

Achieving accurate object tracking against noise spoofing interference caused by similar backgrounds, similar objects, occlusion, or illumination variations is a challenge, especially since Siamese trackers largely ignore the spatial-temporal and part-level information of object structure. To handle this problem, we present a novel object tracking network with spatial-temporal awareness, which learns the correlation of local feature changes and extracts global spatial-temporal fusion features of the object to prevent noise spoofing interference. Specifically, our tracker integrates an object graph reconstruction representation module and a spatial-temporal graph Transformer module. The graph reconstruction representation module models the object structure and search region as part-to-part graph node correspondences, propagating object information to achieve part-level feature aggregation in the search region. The proposed spatial-temporal graph Transformer module fuses the temporal and spatial features of the object to enhance the contextual features and correlation between temporal and spatial feature variations, recognizing the noise spoofing interference to enhance the tracking performance. Meanwhile, our tracker learns object motion information to improve object state awareness, enhancing tracking accuracy under similar objects interference. Extensive experiments on six public datasets validate that our tracker outperforms related state-of-the-art trackers and achieves accurate tracking results against noise spoofing interference.

Computer vision-based displacement measurement using spatio-temporal context and optical flow considering illumination variation

Computer vision-based displacement measurement using spatio-temporal context and optical flow considering illumination variation

The effect of illumination on the visibility of steps and ramps for people with low vision.

Poor visibility of indoor features such as steps and ramps can pose mobility hazards for people with low vision. For purposes of architectural design, it is important to understand how design parameters such as the illumination level of an indoor space affect the visibility of steps and ramps. This study was aimed to examine the effect of typical variation in photopic illumination level in an indoor space on the visibility of steps and ramps for individuals with low vision. Steps and ramps were constructed in a large windowless room illuminated by overhead lights. Subjects with low vision completed a 5-alternative forced choice task to recognize the targets at three levels of photopic illumination, i.e., 800, 80, and 8 lux, and gave confidence ratings about their judgments on a 5-point scale. Acuities and contrast sensitivities of the subjects were also measured at each illumination level. For comparison, a group of normally sighted subjects with simulated acuity reduction also completed the step-and-ramp recognition task. For both groups of subjects, recognition accuracy was not affected by illumination level. For subjects with low vision, however, there was a significant effect of illumination level on confidence rating: subjects became more confident about their judgments with increasing illumination. There was also a weak effect of illumination level on acuity and contrast sensitivity, both worsening with decreasing illumination. Recognition performance was best predicted by contrast sensitivity, whereas confidence was best predicted by visual acuity. Illumination variation over a typical photopic range in an indoor space had minimal effect on the objective visibility of steps and ramps for people with low vision. However, illumination level affected subjects' confidence in hazard recognition. Design decisions on parameters such as illumination should consider the consequences on both the objective and the subjective accessibility of a space.

Is human perception reliable? Toward illumination robust food freshness prediction from food appearance — Taking lettuce freshness evaluation as an example

Machine vision-based food quality evaluation systems have achieved great attention in industry and academia in recent years because of their high-throughput and non-invasive properties. In practice, the environmental illumination conditions variations would cause visual evaluation bias for both human and machine perceptions. Compared to other existing studies which take sample images and human visual grading under fixed illumination conditions, this study first investigated the environmental illumination effects on the performance of visual-based food grading for both humans and machines. Taking lettuce samples as an example, an image dataset that considered the environmental illumination variations was first established. This dataset encompasses human visual grading scores obtained from sensory panels as well. In contrast to current studies that utilize a single grader or L*a*b* color features to assess sample freshness, our study reveals a substantial impact of environmental illumination on both human perception (p<0.0001) and L*a*b* color features (p<0.0001). In order to enhance the performance of machine vision-based freshness prediction, multitask learning protocols were incorporated into the proposed new network architectures. This allowed the simultaneous prediction of both sample freshness and illumination conditions. In comparison to commonly used generic convolutional neural network models and vision transformer models, the newly proposed model exhibited superior freshness prediction performance. It minimized the prediction error by 20.36%, outperforming the generic ResNet model. This research represents the first quantitative study addressing human and machine perceptions under varied illumination conditions for food quality evaluation. The findings are anticipated to play a pivotal role in expediting the integration of machine vision applications into food engineering practices.

A visual area detection algorithm based on mirror splicing for motorcycle indirect vision test

Abstract Motorcycle indirect vision devices are important safety components, and the visual area is required to meet industry standards. There is a need for effective algorithms to detect and analyze the relevant visual areas within these mirrors to enhance motorcycle safety. However, it is a challenge to test rearview mirror vision in unstructured environments given variations in illumination, occlusion, and object scale. We propose a visual area detection algorithm based on mirror stitching for a motorcycle indirect vision test. First, an edge-based VGG16-Unet (EBV16-Unet) network is employed to extract binocular mirror information and eliminate the complex background. Second, gradient-based topology-preserving image stitching and multi-band hybrid Laplacian pyramid-based image blending algorithms are utilized to complete binocular mirror information acquisition. Finally, a sequential detection method for adaptive marker color and shape features is used to establish the visual area. The EBV16-Unet algorithm achieved an accuracy of 98.63% for precision, 98.71% for recall, 98.58% for F1, and 98.37% for mean intersection-over-union (MIOU), surpassing the comparative models of PSPNet, DeepLab v3+, and HRNet and exhibited superior generalization ability. The binocular vision splicing effect experiment revealed a horizontal splicing error of 0.114322 ± 0.0674 and vertical splicing error of 0.124287 ± 0.063302, calculated using a standard checkerboard. The rearview mirror vision test operation experiment results confirm that the Motorcycle Indirect Vision Test System (MIVTS) offers convenience, simplicity and high accuracy. MIVTS successfully accomplishes the unstructured motorcycle rearview mirror vision test, thereby establishing an advanced theoretical foundation for computer vision-based automated vehicle inspection.