Sudden Illumination Changes Research Articles

Automatic exposure strategy network for robust visual odometry in environments with high dynamic range

The success of visual-based localization requires the image features to be sufficiently captured in well-exposed images and reliably tracked among image sequences. In this paper, a learning-based exposure strategy algorithm is developed to improve the performance of visual odometry or simultaneous localization and mapping in scenes with sudden illumination changes and high dynamic range (HDR). The exposure value is predicted by optimizing a novel image information metric, which is calculated based on a multi-scale image histogram. Experimental observations show that the proposed metric is highly related to the localization performance, and achieves better localization accuracy when compared to other state-of-the-art metrics. Using this novel metric, the exposure control network is designed to predict the exposure value for the next image given a sequence of recently captured images as inputs. For different application scenarios, alternative network frameworks are proposed with the options to include an optimization module providing improved exposure values for feedback learning, but with additional computation complexity. The approach is light-weight for real-time video applications and the images captured using the predicted exposure values are well-exposed with better detected and matched features in a variety of challenging HDR scenes. Code will be available in https://github.com/leejieyu13/HDR-AE.

Research on trajectory control of multi‐degree‐of‐freedom industrial robot based on visual image

AbstractIn order to improve the trajectory control effect of multi‐degree‐of‐freedom industrial robots, this paper combines visual image technology to conduct research on trajectory control of multi‐degree‐of‐freedom industrial robots. Aiming at the problem of video segmentation under sudden illumination changes, this paper uses a Gaussian mixture model based on the global illumination function to adopt a variety of illumination invariant features, and proposes a scene segmentation algorithm suitable for sudden illumination changes. Moreover, this paper compares and verifies the algorithm from the subjective and objective perspectives through experiments, which shows that the algorithm in this paper can segment the scene more accurately even in the environment of sudden changes in illumination. In addition, the results of the accuracy test and the trajectory control test show that the research method of the multi‐degree‐of‐freedom industrial robot trajectory control based on the visual image proposed in this paper can effectively improve the trajectory control effect of the robot.

Accurate voltage regulation for a DC microgrid using nonlinear state feedback controller

AbstractThe high penetration of distributed generation systems poses challenges in effectively managing both DC bus voltage and power‐sharing in DC microgrids (MGs). To ensure precise voltage regulation across various distributed generation systems and maintain overall system stability, this paper studies the modelling and control of an islanded DC MG described by a set of nonlinear‐nonaffine differential equations. In the proposed DC MG, a battery storage, a photovoltaic (PV) power plant, and a back‐end power converter with DC loads have been connected to a common DC bus. First, a dynamic model of a DC MG was developed. Then, to find the maximum power point of solar PV, a straightforward method based on an artificial neural net is employed. Afterward, a nonlinear local state feedback controller through output regulation theory is applied to achieve voltage stability, overcoming the drawbacks of droop‐based techniques. Moreover, partial state feedback capability simplifies controller design. To verify the effectiveness of the proposed approach, various scenarios, including sudden load variation, illumination change, and changes in the MG configuration, have been examined. Finally, numerical studies were carried out in a simulation environment to demonstrate the superiority of the proposed nonlinear control scheme over constant feedback control.

Improvement and Simulation of PV Power Generation Control Algorithm Based on MPPT

As the traditional PV MPPT algorithm cannot meet both the tracking speed and steady-state accuracy, a new solution idea is provided to improve the traditional conductivity increment method on basis of the slope of the PV cell output P-V curve. The algorithm dynamically changes the step size by combining duty cycle and power and exponential functions, and rapidly adapts to variation in the external environment, solving the slow tracking speed and low steady-state accuracy when the traditional conductance incremental method is subject to sudden changes in illumination. The emulation results indicate that the modified algorithm and the alterable-step increment conductivity method can be simulated in the same environment using MATLAB/Simulink software. The emulation results indicate that the modified algorithm, compared with the traditional algorithm in tracking the MPPT speed and amplitude, can effectively reduce the tracking time with high steady-state accuracy.

SemAttNet: Toward Attention-Based Semantic Aware Guided Depth Completion

Depth completion involves recovering a dense depth map from a sparse map and an RGB image. Recent approaches focus on utilizing color images as guidance images to recover depth at invalid pixels. However, color images alone are not enough to provide the necessary semantic understanding of the scene. Consequently, the depth completion task suffers from sudden illumination changes in RGB images (e.g., shadows). In this paper, we propose a novel three-branch backbone comprising color-guided, semantic-guided, and depth-guided branches. Specifically, the color-guided branch takes a sparse depth map and RGB image as an input and generates color depth which includes color cues (e.g., object boundaries) of the scene. The predicted dense depth map of color-guided branch along-with semantic image and sparse depth map is passed as input to semantic-guided branch for estimating semantic depth. The depth-guided branch takes sparse, color, and semantic depths to generate the dense depth map. The color depth, semantic depth, and guided depth are adaptively fused to produce the output of our proposed three-branch backbone. In addition, we also propose to apply semantic-aware multi-modal attention-based fusion block (SAMMAFB) to fuse features between all three branches. We further use CSPN++ with Atrous convolutions to refine the dense depth map produced by our three-branch backbone. Extensive experiments show that our model achieves state-of-the-art performance in the KITTI depth completion benchmark at the time of submission.

Temporal Head Pose Estimation From Point Cloud in Naturalistic Driving Conditions

Head pose estimation is an important problem as it facilitates tasks such as gaze estimation and attention modeling. In the automotive context, head pose provides crucial information about the driver’s mental state, including drowsiness, distraction and attention. It can also be used for interaction with in-vehicle infotainment systems. While computer vision algorithms using RGB cameras are reliable in controlled environments, head pose estimation is a challenging problem in the car due to sudden illumination changes, occlusions and large head rotations that are common in a vehicle. These issues can be partially alleviated by using depth cameras. Head rotation trajectories are continuous with important temporal dependencies. Our study leverages this observation, proposing a novel temporal deep learning model for head pose estimation from point cloud. The approach extracts discriminative feature representation directly from point cloud data, leveraging the 3D spatial structure of the face. The frame-based representations are then combined with <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">bidirectional long short term memory</i> (BLSTM) layers. We train this model on the newly collected <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">multimodal driver monitoring</i> (MDM) dataset, achieving better results compared to non-temporal algorithms using point cloud data, and state-of-the-art models using RGB images. We further show quantitatively and qualitatively that incorporating temporal information provides large improvements not only in accuracy, but also in the smoothness of the predictions.

Video Inpainting: A Complete Framework

Reconstructing and repairing of corrupted or missing parts after object removal in digital video is an important trend in artwork restoration. Video inpainting is an active subject in video processing, which deals with the recovery of the corrupted or missing data. Most previous video inpainting approaches consume more time in extensive search to find the best patch to restore the damaged frames. In addition to that, most of them cannot handle the gradual and sudden illumination changes, dynamic background, full object occlusion, and object changes in scale. In this paper, we present a complete video inpainting framework without the extensive search process. The proposed framework consists of a segmentation stage based on low-resolution version and background subtraction. A background inpainting stage is applied to restore the damaged background regions after static or moving object removal based on the gray-level co-occurrence matrix (GLCM). A foreground inpainting stage is based on objects repository. GLCM is used to complete the moving occluded objects during the occlusion. The proposed method reduces the inpainting time from hours to a few seconds and maintains the spatial and temporal consistency. It works well when the background has clutter or fake motion, and it can handle the changes in object size and in posture. Moreover, it is able to handle full occlusion and illumination changes.

End-to-End Recurrent Generative Adversarial Network for Traffic and Surveillance Applications

In video frame segmentation, many existing deep networks and contemporary approaches give a remarkable performance with the assumption that the only foreground is moving, and the background is stationary. However, in the presence of infrequent motion of foreground objects, sudden illumination changes in the background, bad weather, and dynamic background, the accurate foreground object(s) segmentation is a challenging task. Generative adversarial networks (GAN) based training shows fruitful results in various fields like image-to-image style transfer, image enhancement, semantic segmentation, image super-resolution, etc. The limited results of hand-crafted approaches for moving object segmentation (MOS) and the robustness of adversarial training for a given task inspired us to propose a novel approach for moving object segmentation (MOS). In this context, an end-to-end generative adversarial network (two generators) with recurrent technique is proposed for MOS and is named as RMS-GAN. The proposed RMS-GAN is able to incorporate foreground probability knowledge with residual and weight sharing based recurrent technique for accurate segmentation. The recurrent technique helps us to exhibit the temporal behavior between successive video frames, which is more prominent for any video processing applications. Also, to enhance the spatial coherence of the obtained foreground probability map using the generator-1 network, the cascaded architecture of two generators is proposed. The effectiveness of the proposed approach is evaluated both qualitatively and quantitatively on three benchmark video datasets for MOS. Experimental result analysis shows that the proposed network outperforms the existing state-of-the-art methods on three benchmark datasets for MOS.

ROVINS: Robust Omnidirectional Visual Inertial Navigation System

Visual odometry is an essential component in robot navigation and autonomous driving; however, visual sensors are vulnerable in fast motion or sudden illumination changes. This weakness can be compensated with inertial measurement units (IMUs), which maintain the short-term motion when visual sensing becomes unstable and enhance the quality of estimated motion with inertial information. An omnidirectional multi-view visual odometry (ROVO) has recently demonstrated superior performance and stability with the unceasing feature observation of the omnidirectional setup; however, the shortcomings of visual odometry remain. This letter introduced an omnidirectional visual-inertial odometry system (ROVINS) that could seamlessly integrate the inertial information into the omnidirectional visual odometer algorithm: (a) The soft relative pose constraints from the inertial measurement are first added to the pose optimization formulation, which enables blind motion estimation when all visual features are lost; (b) Using the prediction results from the estimated velocity, the visual features in tracking are initialized, resulting in feature tracking that is more robust to visual disturbances. The experimental results showed that the proposed ROVINS algorithm outperforms the vision-only algorithm by a significant margin.

Fast Visual Tracking Using Spatial Temporal Background Context Learning

Visual Tracking by now has gained much provenience among researchers in recent years due to its vast variety of applications that occur in daily life. Various applications of visual tracking include counting of cars on a high way, analyzing the crowd intensity in a concert or a football ground or a surveillance camera tracking a single person to track its movements. Various techniques have been proposed and implemented in this research domain where researchers have analyzed various parameters. Still this area has a lot to offer. There are two common approaches that are currently deployed in visual tracking. One is discriminative tracking and the other one is generative tracking. Discriminative tracking requires a pre-trained model that requires the learning of the data and solves the object recognition as a binary classification problem. On the other hand, generative model in tracking makes use of the previous states so that next state can be predicted. In this paper, a novel tacking based on generative tracking method is proposed called as Illumination Inavariant Spatio Temporal Tracker (IISTC). The proposed technique takes into account of the nearby surrounding regions and performs context learning so that the state of the object under consideration and its surrounding regions can be estimated in the next frame. The learning model is deployed both in the spatial domain as well as the temporal domain. Spatial domain part of the tracker takes into consideration the nearby pixels in a frame while the temporal model takes account of the possible change of object location. The proposed tracker was tested on a set of 50 images against other state of the art four trackers. Experimental results reveal that our proposed tracker performs reasonably well as compared with other trackers. The proposed visual tracker is both efficiently with respect to computation power as well as accuracy. The proposed tracker takes only 4 fast Fourier transform computations thus making it reasonably faster. The proposed trackers perform exceptionally well when there is a sudden change in back ground illumination.

Image editing-based data augmentation for illumination-insensitive background subtraction

PurposeA core challenge in background subtraction (BGS) is handling videos with sudden illumination changes in consecutive frames. In our pilot study published in, Sakkos:SKIMA 2019, we tackle the problem from a data point-of-view using data augmentation. Our method performs data augmentation that not only creates endless data on the fly but also features semantic transformations of illumination which enhance the generalisation of the model.Design/methodology/approachIn our pilot study published in SKIMA 2019, the proposed framework successfully simulates flashes and shadows by applying the Euclidean distance transform over a binary mask generated randomly. In this paper, we further enhance the data augmentation framework by proposing new variations in image appearance both locally and globally.FindingsExperimental results demonstrate the contribution of the synthetics in the ability of the models to perform BGS even when significant illumination changes take place.Originality/valueSuch data augmentation allows us to effectively train an illumination-invariant deep learning model for BGS. We further propose a post-processing method that removes noise from the output binary map of segmentation, resulting in a cleaner, more accurate segmentation map that can generalise to multiple scenes of different conditions. We show that it is possible to train deep learning models even with very limited training samples. The source code of the project is made publicly available at https://github.com/dksakkos/illumination_augmentation

Texture collinearity foreground segmentation for night videos

One of the most difficult scenarios for unsupervised segmentation of moving objects is found in nighttime videos where the main challenges are the poor illumination conditions resulting in low-visibility of objects, very strong lights, surface-reflected light, a great variance of light intensity, sudden illumination changes, hard shadows, camouflaged objects, and noise. This paper proposes a novel method, coined COLBMOG (COLlinearity Boosted MOG), devised specifically for the foreground segmentation in nighttime videos, that shows the ability to overcome some of the limitations of state-of-the-art methods and still perform well in daytime scenarios. It is a texture-based classification method, using local texture modeling, complemented by a color-based classification method. The local texture at the pixel neighborhood is modeled as an N-dimensional vector. For a given pixel, the classification is based on the collinearity between this feature in the input frame and the reference background frame. For this purpose, a multimodal temporal model of the collinearity between texture vectors of background pixels is maintained. COLBMOG was objectively evaluated using the ChangeDetection.net (CDnet) 2014, Night Videos category, benchmark. COLBMOG ranks first among all the unsupervised methods. A detailed analysis of the results revealed the superior performance of the proposed method compared to the best performing state-of-the-art methods in this category, particularly evident in the presence of the most complex situations where all the algorithms tend to fail.

An Objective Image Quality Evaluation and its Applications for Low Illumination and Sudden Illumination Changes

An Objective Image Quality Evaluation and its Applications for Low Illumination and Sudden Illumination Changes

Robust Vehicle Detection and Counting Algorithm Adapted to Complex Traffic Environments with Sudden Illumination Changes and Shadows.

The real-time vehicle detection and counting plays a crucial role in traffic control. To collect traffic information continuously, the access to information from traffic video shows great importance and huge advantages compared with traditional technologies. However, most current algorithms are not adapted to the effects of undesirable environments, such as sudden changes in illumination, vehicle shadows, and complex urban traffic conditions, etc. To address these problems, a new vehicle detection and counting method was proposed in this paper. Based on a real-time background model, the problem of sudden illumination changes could be solved, while the vehicle shadows could be removed using a detection method based on motion. The vehicle counting was built on two types of ROIs—called Normative-Lane and Non-Normative-Lane—which could adapt to the complex urban traffic conditions, especially for non-normative driving. Results have shown that the methodology we proposed is able to count vehicles with 99.93% accuracy under the undesirable environments mentioned above. At the same time, the setting of the Normative-Lane and the Non-Normative-Lane can realize the detection of non-normative driving, and it is of great significance to improve the counting accuracy.

Proposing a new feature descriptor for moving object detection

Segmenting moving objects from various video streams especially in the presence of complex dynamic backgrounds is one of the most pivotal and fundamental tasks in computer vision. Although there are many high quality proposals for moving object detection and segmentation, still some issues such as sudden changes in illumination, camera jitter, swaying vegetation, rippling water, etc need to be resolved. This paper proposes a new method for moving object detection using background subtraction technique. To this end, a new color feature descriptor and a new model updating strategy are proposed for ameliorating the effects of these natural challenges. The proposed descriptor is a local fuzzy descriptor that tries to extract the color feature of a pixel by looking at its neighbors. Because of its local nature, this descriptor is able to accurately detect and extract moving objects. The fuzzy nature of the descriptor helps it to be robust against diverse challenging situations. In order to validate the functionality and show the supremacy of the proposed method, comprehensive experiments were conducted on various challenging datasets. From the experimental results, it is observed that the proposed method achieves acceptable results for moving object detection and segmentation.

The driver’s visual perception research to analyze pedestrian safety at twilight

Road traffic movement at nightfall (twilight) is characterizing by a reduction of light time of the day and the rapid nightfall onset, thus the driver’s eyes have less time to adapt to rapid sudden changes in illumination. The visual perception and the reaction time of the driver in conditions when pedestrians appear in nightfall conditions on the street and road network in a city is considered in the paper. Researched was conducted in uncontrolled pedestrian crossings in nightfall conditions on Ukrainian roads. Regularities of the vehicle’s driver and pedestrians’ interaction in nightfall conditions are obtained. Road traffic accidents occurrence probabilities at the twilight time considering the driver’s reaction time and the car’s movement parameters was analyzed. As a result, a model for estimating the variation the reaction time of the driver when a pedestrian appears in the nightfall (twilight) conditions was calibrated.

Robust Detection of Abandoned Object for Smart Video Surveillance in Illumination Changes.

Most existing abandoned object detection algorithms use foreground information generated from background models. Detection using the background subtraction technique performs well under normal circumstances. However, it has a significant problem where the foreground information is gradually absorbed into the background as time passes and disappears, making it very vulnerable to sudden illumination changes that increase the false alarm rate. This paper presents an algorithm for detecting abandoned objects using a dual background model, which is robust even in illumination changes as well as other complex circumstances like occlusion, long-term abandonment, and owner re-attendance. The proposed algorithm can adapt quickly to various illumination changes. And also, it can precisely track the target objects to determine whether it is abandoned regardless of the existence of foreground information and the effect from the illumination changes, thanks to the largest-contour-based presence authentication mechanism proposed in this paper. For performance evaluation, we trialed the algorithm with the PETS2006, ABODA datasets as well as our dataset, especially to demonstrate its robustness in various illumination changes.

Vehicle Counting in Video Sequences: An Incremental Subspace Learning Approach.

The counting of vehicles plays an important role in measuring the behavior patterns of traffic flow in cities, as streets and avenues can get crowded easily. To address this problem, some Intelligent Transport Systems (ITSs) have been implemented in order to count vehicles with already established video surveillance infrastructure. With this in mind, in this paper, we present an on-line learning methodology for counting vehicles in video sequences based on Incremental Principal Component Analysis (Incremental PCA). This incremental learning method allows us to identify the maximum variability (i.e., motion detection) between a previous block of frames and the actual one by using only the first projected eigenvector. Once the projected image is obtained, we apply dynamic thresholding to perform image binarization. Then, a series of post-processing steps are applied to enhance the binary image containing the objects in motion. Finally, we count the number of vehicles by implementing a virtual detection line in each of the road lanes. These lines determine the instants where the vehicles pass completely through them. Results show that our proposed methodology is able to count vehicles with 96.6% accuracy at 26 frames per second on average—dealing with both camera jitter and sudden illumination changes caused by the environment and the camera auto exposure.

Enhanced Median Flow Tracker Based on Photometric Correction for Videos with Abrupt Changing Illumination

Object tracking is a fundamental task in video surveillance, human-computer interaction and activity analysis. One of the common challenges in visual object tracking is illumination variation. A large number of methods for tracking have been proposed over the recent years, and median flow tracker is one of them which can handle various challenges. Median flow tracker is designed to track an object using Lucas-Kanade optical flow method which is sensitive to illumination variation, hence fails when sudden illumination changes occur between the frames. In this paper, we propose an enhanced median flow tracker to achieve an illumination invariance to abruptly varying lighting conditions. In this approach, illumination variation is compensated by modifying the Discrete Cosine Transform (DCT) coefficients of an image in the logarithmic domain. The illumination variations are mainly reflected in the low-frequency coefficients of an image. Therefore, a fixed number of DCT coefficients are ignored. Moreover, the Discrete Cosine (DC) coefficient is maintained almost constant all through the video based on entropy difference to minimize the sudden variations of lighting impacts. In addition, each video frame is enhanced by employing pixel transformation technique that improves the contrast of dull images based on probability distribution of pixels. The proposed scheme can effectively handle the gradual and abrupt changes in the illumination of the object. The experiments are conducted on fast-changing illumination videos, and results show that the proposed method improves median flow tracker with outperforming accuracy compared to the state-of-the-art trackers

New Insights of Background Estimation and Region Localization

Subtraction of background in a crowded scene is a crucial and challenging task of monitoring the surveillance systems. Because of the similarity between the foreground object and the background, it is known that the background detection and moving foreground objects is difficult. Most of the previous works emphasize this field but they cannot distinguish the foreground from background due to the challenges of gradual or sudden illumination changes, high-frequencies background objects of motion changes, background geometry changes and noise. After getting the foreground objects, segmentation is need to localize the objects region. Image segmentation is a useful tool in many areas, such as object recognition, image processing, medical image analysis, 3D reconstruction, etc. In order to provide a reliable foreground image, a carefully estimated background model is needed. To tackle the issues of illumination changes and motion changes, this paper establishes an effective new insight of background subtraction and segmentation that accurately detect and segment the foreground people. The scene background is investigates by a new insight, namely Mean Subtraction Background Estimation (MS), which identifies and modifies the pixels extracted from the difference of the background and the current frame. Unlike other works, the first frame is calculated by MS instead of taking the first frame as an initial background. Then, this paper make the foreground segmentation in the noisy scene by foreground detection and then localize these detected areas by analyzing various segmentation methods. Calculation experiments on the challenging public crowd counting dataset achieve the best accuracy than state-of-the-art results. This indicates the effectiveness of the proposed work.