Brightness Constancy Assumption Research Articles

Rationalizing Efficient Compositional Image Alignment

We study the issue of computational efficiency for Gauss-Newton (GN) non-linear least-squares optimization in the context of image alignment. We introduce the Constant Jacobian Gauss-Newton (CJGN) optimization, a GN scheme with constant Jacobian and Hessian matrices, and the equivalence and independence conditions as the necessary requirements that any function of residuals must satisfy to be optimized with this efficient approach. We prove that the Inverse Compositional (IC) image alignment algorithm is an instance of a CJGN scheme and formally derive the compositional and extended brightness constancy assumptions as the necessary requirements that must be satisfied by any image alignment problem so it can be solved with an efficient compositional scheme. Moreover, in contradiction with previous results, we also prove that the forward and inverse compositional algorithms are not equivalent. They are equivalent, however, when the extended brightness constancy assumption is satisfied. To analyze the impact of the satisfaction of these requirements we introduce a new image alignment evaluation framework and the concepts of short- and wide-baseline Jacobian. In wide-baseline Jacobian problems the optimization will diverge if the requirements are not satisfied. However, with a good initialization, a short-baseline Jacobian problem may converge even if the requirements are not satisfied.

Illumination-Robust Optical Flow Using a Local Directional Pattern

Most of the variational optical flow methods are based on the well-known brightness constancy assumption or high-order constancy assumptions to implement the data term in the optimization energy function. Unfortunately, any variation in the lighting within the scene violates the brightness constancy constraint; in turn, the gradient constancy assumption does not work properly with large illumination changes. This paper proposes an illumination-robust constancy based on a robust texture descriptor rather than the brightness constancy. Thus, the similarity function used as a data term was obtained from extracting texture features through the local directional pattern descriptor for two consecutive frames within the duality total variational optical flow algorithm. In addition, a weighted nonlocal term that depends on both the color similarity and the occlusion state of pixels is integrated during the optimization process to increase the accuracy of the resulting flow field. The experimental results show a qualitative comparison with the proposed approach and yield state-of-the-art results on the KITTI, Midleburry, and MPI-sintel data sets.

Robust Optical Flow Estimation

In this work, we describe an implementation of the variational method proposed by Brox et al. in 2004, which yields accurate optical flows with low running times. It has several benefits with respect to the method of Horn and Schunck: it is more robust to the presence of outliers, produces piecewise-smooth flow fields and can cope with constant brightness changes. This method relies on the brightness and gradient constancy assumptions, using the information of the image intensities and the image gradients to find correspondences. It also generalizes the use of continuous L 1 functionals, which help mitigate the effect of outliers and create a Total Variation (TV) regularization. Additionally, it introduces a simple temporal regularization scheme that enforces a continuous temporal coherence of the flow fields. Source Code The source code, the code documentation, and the online demo are accessible at the IPOL web page of this article 1 . In this page an implementation is available for download. This file contains two directories: one for the spatial method and another for the temporal method. The spatial method is suitable for general image sequences, while the temporal method should be used when the flow fields are known to be very continuous.

Revisiting Lucas-Kanade and Horn-Schunck

This paper revisits classical Lucas-Kanade (LK) and Horn-Schunck (HS) optical flow techniques. The aim is to provide a baseline for other researchers on these two timeless techniques. The formulations presented incorporate modern practices, namely multichannel, multi-resolution with refinement procedure (warping), non-quadratic penalisers and non-linear formulation of the brightness assumption. The experiments conducted demonstrate the performance enhancement that can be assigned to each modern practice. Thereby, the performance of the LK and HS is renewed in order to enable a fair comparison to other state-of-art techniques. The good performance of today's optical flow formulations is possible due to several modern practices, namely multi- resolution (coarse-to-fine) refinement that deals with large displacements, iterative methods based on warping interpolations, multi-channels, robustification of each channel (non- quadratic penalisers), gradient constancy assumption to handle small variation of the brightness value, and photometric colour spaces. Usually, the performance of state-of-art techniques is compared to the classical Lucas-Kanade (LK) and Horn-Schunck (HS) presented by Barron et al. (1994) (2) . We consider these comparisons unfair because the LK and HS do not contemplate the most important modern practices (such as, multi-resolution with refinement, nonlinear formulation of the brightness constancy assumption). Therefore, the full potential of both techniques is not considered. For that reason, this article intends to provide a modern formulation of the LK and HS that incorporates multi-channel, multi-resolution with refinement (iterative procedure based on warping) and nonlinear brightness assumption. In addition, non-quadratic penaliser based on the Charbonnier error function is considered for the neighbourhood weighting of the LK. Thereby, the contributions of this paper include:  Modern and coloured formulation of the LK and HS  Extensive qualitative and quantitative evaluation  Baseline performance analysis for other research The experimental analysis includes comparisons of both non-hierarchical and hierarchical pyramidal architectures. The contribution of the iterative refinement and multi-channel approach is also studied. Finally, the neighbourhood weights of the LK are iteratively defined using the non-quadratic penaliser. When implemented with appropriate and modern practices, classical optical flow formulations such as Lucas- Kanade (9) and Horn-Schunck (10) can achieve respectable optical flow estimations.

Myocardial Motion Estimation From Medical Images Using the Monogenic Signal

We present a method for the analysis of heart motion from medical images. The algorithm exploits monogenic signal theory, recently introduced as an N-dimensional generalization of the analytic signal. The displacement is computed locally by assuming the conservation of the monogenic phase over time. A local affine displacement model is considered to account for typical heart motions as contraction/expansion and shear. A coarse-to-fine B-spline scheme allows a robust and effective computation of the model's parameters, and a pyramidal refinement scheme helps to handle large motions. Robustness against noise is increased by replacing the standard point-wise computation of the monogenic orientation with a robust least-squares orientation estimate. Given its general formulation, the algorithm is well suited for images from different modalities, in particular for those cases where time variant changes of local intensity invalidate the standard brightness constancy assumption. This paper evaluates the method's feasibility on two emblematic cases: cardiac tagged magnetic resonance and cardiac ultrasound. In order to quantify the performance of the proposed method, we made use of realistic synthetic sequences from both modalities for which the benchmark motion is known. A comparison is presented with state-of-the-art methods for cardiac motion analysis. On the data considered, these conventional approaches are outperformed by the proposed algorithm. A recent global optical-flow estimation algorithm based on the monogenic curvature tensor is also considered in the comparison. With respect to the latter, the proposed framework provides, along with higher accuracy, superior robustness to noise and a considerably shorter computation time.

Robust disparity estimation based on color monogenic curvature phase

Disparity estimation for binocular images is an important problem for many visual tasks such as 3D environment reconstruction, digital hologram, virtual reality, robot navigation, etc. Conventional approaches are based on brightness constancy assumption to establish spatial correspondences between a pair of images. However, in the presence of large illumination variation and serious noisy contamination, conventional approaches fail to generate accurate disparity maps. To have robust disparity estimation in these situations, we first propose a model - color monogenic curvature phase to describe local features of color images by embedding the monogenic curvature signal into the quaternion representation. Then a multiscale framework to estimate disparities is proposed by coupling the advantages of the color monogenic curvature phase and mutual information. Both indoor and outdoor images with large brightness variation are used in the experiments, and the results demonstrate that our approach can achieve a good performance even in the conditions of large illumination change and serious noisy contamination.

Multi-view Scene Flow Estimation: A View Centered Variational Approach

We present a novel method for recovering the 3D structure and scene flow from calibrated multi-view sequences. We propose a 3D point cloud parametrization of the 3D structure and scene flow that allows us to directly estimate the desired unknowns. A unified global energy functional is proposed to incorporate the information from the available sequences and simultaneously recover both depth and scene flow. The functional enforces multi-view geometric consistency and imposes brightness constancy and piecewise smoothness assumptions directly on the 3D unknowns. It inherently handles the challenges of discontinuities, occlusions, and large displacements. The main contribution of this work is the fusion of a 3D representation and an advanced variational framework that directly uses the available multi-view information. This formulation allows us to advantageously bind the 3D unknowns in time and space. Different from optical flow and disparity, the proposed method results in a nonlinear mapping between the images’ coordinates, thus giving rise to additional challenges in the optimization process. Our experiments on real and synthetic data demonstrate that the proposed method successfully recovers the 3D structure and scene flow despite the complicated nonconvex optimization problem.

Stripe nonuniformity correction based on registration for infrared-focal plane arrays

Stripe nonuniformity is a very typical fixed pattern noise in line infrared focal plane arrays(IR-FPA) and uncooled staring IR-FPA.Its origination was analyzed.A stripe nonuniformity correction algorithm based on brightness constancy assumption and registration was proposed.Brightness mean-square error function was obtained according to adjacent frames.Nonuniformity correction parameters could be derived through the unique global optimum by minimizing the function.Experimental results indicate that the proposed algorithm could achieve a great convergence in several frames with low computational complexity and enhance the effective of stripe nonuniformity correction.

Scene based nonuniformity correction based on block ergodicity for infrared focal plane arrays

This paper puts forward a new scene based nonuniformity correction algorithm for IRFPA. This method adopts phase-correlation method for motion estimation and takes the sum of mean-square errors of the pixel brightness between several adjacent frames as the cost function when the brightness constancy assumption between two adjacent frames is satisfied. Nonuniformity correction parameters could be estimated by minimizing such cost function. In order to reduce calculation quantity, we can divide these images into several subblocks, and solve for the optimum solution of the cost function respectively in each subblock. From the analysis, it is shown that the optimum solution is of global uniqueness when all the elements in subblocks could satisfy the ergodicity condition. Then the estimated value of nonuniformity correction parameters could be deduced by minimizing the cost functions. The nonuniformity correction experiments for both infrared image sequence with simulated nonuniformity and infrared imagery with real nonuniformity show that the proposed algorithm could achieve a great correction effect by only analyzing a small number of frames.

Smoothness Constraints in Recursive Search Motion Estimation for Picture Rate Conversion

Many motion compensation algorithms are based on block matching. The quality of the block correlation depends on the validity of the brightness constancy assumption and the assumption of fixed translational motion within a block. These assumptions are invalid in areas with texture changes, noise, lighting changes, and rapid deformations. Smoothness priors should enforce stable estimates in these regions by propagating neighboring estimates, while preserving hard object boundaries (piecewise smoothness). Most motion estimation algorithms that successfully implement these constraints are computationally complex. In this paper, we show an intuitive and computationally efficient way to implement them within the framework of (real-time) recursive search, targeting consumer-market embedded devices with limited resources.

Identification of velocity fields for geophysical fluids from a sequence of images

We propose an algorithm to estimate the motion between two images. This algorithm is based on the nonlinear brightness constancy assumption. The number of unknowns is reduced by considering displacement fields that are piecewise linear with respect to each space variable, and the Jacobian matrix of the cost function to be minimized is assembled rapidly using a finite-element method. Different regularization terms are considered, and a multiscale approach provides fast and efficient convergence properties. Several numerical results of this algorithm on simulated and experimental geophysical flows are presented and discussed.

Low dimensional optimization for in vivo real-time porcine liver motion estimation using ultrasound imaging

This study presents a contribution to the tracking of a moving target during high-intensity focused ultrasound (HIFU) treatment. Indeed, HIFU has proved to be highly efficient in inducing homogeneous and reproducible tumor destruction by thermal coagulation necrosis. However, accurate targeting of human abdominal tumors is difficult to maintain due to the motion induced by breathing. An algorithm is presented to track a region of interest of fixed size in a sequence of images. This algorithm was evaluated on synthetic data and on in vivo sequences of ultrasound liver images acquired using 12 MHz ultrasound imaging probe at a rate of 16 frames/s. The algorithm presented here was derived from the non-linear constant brightness assumption. Since the motion was smooth it was possible to reduce the space of admissible displacements; hence the number of unknown parameters was small compared with the size of the data. The optimal displacement was estimated by a Gauss–Newton method, and the matrix required at each step was assembled by reading the data only once. This algorithm was applied to simulated data, where the true displacement was known and a precise evaluation was possible. The relative error was about 2%. The algorithm was also applied to a video sequence of sonograms acquired during in vivo experiments. These trials were conducted on porcine liver since its size and physiology are similar to humans. Movements were induced by breathing and heart-beating. Two particular frequencies representing breathing (0.26 Hz) and heart beat (1.14 Hz) were identified in the estimated displacement and were correlated with the monitored breathing (0.27 Hz) and electrocardiograms (1.28 Hz). In addition, a region of interest (ROI) modeling the focal zone of a HIFU transducer was tracked along time. Therefore this study provides a mean of determining the location of the targeted region in vivo during HIFU treatments. This can be applied to correct the location of the focal zone accordingly. This method can preferentially be applied to the liver or to any other moving organ.

Generalized least squares-based parametric motion estimation

The estimation of parametric global motion had a significant attention during the last two decades, but despite the great efforts invested, there are still open issues. The most important ones are related to the accuracy of the estimation and to the ability to recover large deformation between images. In this paper, a new generalized least squares-based motion estimator is proposed. The non-linear Brightness Constancy Assumption is directly used instead of using the classical approach by linearizing the minimization problem using the optical flow equation. In addition, the proposed formulation of the motion estimation problem provides an additional constraint that helps to match the pixels by using the image gradient in the matching process. That is achieved by means of a weight for each observation, assigning high weight values to the observations considered as inliers, i.e. the ones that support the motion model, and low values to the ones considered as outliers. The accuracy of our approach has been tested using challenging real images using both affine and projective motion models. Two motion estimator techniques that uses iteratively reweighted least squares-based (IRLS) techniques to deal with outliers, have been selected for comparison purposes. The results obtained show that the proposed motion estimator can obtain, in most cases, more accurate estimates that the IRLS-based techniques.

Dynamic Texture Detection Based on Motion Analysis

Motion estimation is usually based on the brightness constancy assumption. This assumption holds well for rigid objects with a Lambertian surface, but it is less appropriate for fluid and gaseous materials. For these materials an alternative assumption is required. This work examines three possible alternatives: gradient constancy, color constancy and brightness conservation (under this assumption the brightness of an object can diffuse to its neighborhood). Brightness conservation and color constancy are found to be adequate models. We propose a method for detecting regions of dynamic texture in image sequences. Accurate segmentation into regions of static and dynamic texture is achieved using a level set scheme. The level set function separates each image into regions that obey brightness constancy and regions that obey the alternative assumption. We show that the method can be simplified to obtain a less robust but fast algorithm, capable of real-time performance. Experimental results demonstrate accurate segmentation by the full level set scheme, as well as by the simplified method. The experiments included challenging image sequences, in which color or geometry cues by themselves would be insufficient.

A fast algorithm for image registration

An algorithm is presented here to estimate a smooth motion at a high frame rate. It is derived from the non-linear constant brightness assumption. A hierarchical approach reduces the dimension of the space of admissible displacements, hence the number of unknown parameters is small compared to the size of the data. The optimal displacement is estimated by a Gauss–Newton method, and the matrix required at each step is assembled rapidly using a finite-element method. To cite this article: J. Fehrenbach, M. Masmoudi, C. R. Acad. Sci. Paris, Ser. I 346 (2008).

Epiflow—A paradigm for tracking stereo correspondences

We propose the epiflow as a novel framework to integrate the motion and stereo epipolar geometries for the tracking of stereo matches. For a given stereo pair, spatio-temporal correspondences in the next stereo pair are established from the solution of an optimization problem that incorporates the epiflow geometrical constraint with the brightness constancy assumption as the photometric similarity measure. Two different implementations are proposed, each utilizing the stereo RANSAC algorithm as a mechanism to gain robustness and efficiency in handling large temporal disparity ranges, identify outliers and estimate the motion of the stereo rig as complementary information. The performance of the proposed methods is demonstrated in experiments with various complex underwater images, and comparison with results from the SIFT method allows the assessment of our novel contributions.

Integrating BC & GC Models In Computing Stereo Disparity As Markov Random Field

Belief propagation and graph cuts have emerged as powerful tools for computing efficient approximate so- lution to stereo disparity field modelled as the Markov random field (MRF). These algorithms have provided the best performance based on results on a standard data set (1). However, employment of the brightness constancy (BC) assumption severely limits the range of their applications. Previously, augmenting the BC with gradient constancy (GC) assumption has shown to produce a more robust optical flow algorithm (2), (3). In this paper, these constraints are integrated within the MRF framework to devise an en- hanced global method that broadens the application domains for stereo computation. Results of experiments with both semi-synthetic data and more challenging ocean images are presented to illustrate that the proposed method generally outperforms earlier dense optical flow and stereo algorithms.

Computation of optical flow under non-uniform brightness variations

In this paper, we present a new motion estimation algorithm that provides accurate optical flow computation under non-uniform brightness variations. The proposed algorithm is based on a regularization formulation that minimizes a combination of a modified data constraint energy and a smoothness measure all over the image domain. The data constraint is derived from the conservation of the Laplacian-of-Gaussian (LoG) filtered image function, which alleviates the problem with the traditional brightness constancy assumption under non-uniform illumination variations. The resulting energy minimization is accomplished by an incomplete Cholesky preconditioned conjugate gradient algorithm. Comparisons of experimental results on benchmarking image sequences by using the proposed algorithm and some of the best existing methods are given to demonstrate its superior performance.

A robust multiscale B-spline function decomposition for estimating motion transparency

Motion transparency phenomena in image sequences are frequent, but classical methods of motion estimation are unable to deal with them. This paper describes a method for estimating optical flow by a generalization of the brightness constancy assumption to additive transparencies. The brightness constancy assumption is obtained by setting constant velocity fields during three images of a sequence. Thus, by a Taylor development to its second order, we reach an extension of the optical flow constraint equation. Since the equation is nonlinear, the Levenberg-Marquardt algorithm is used. In order to suppress the unavoidable aperture problem, a global model based on B-spline basis functions is applied with the aim of constraining optical flows. This description of motion allows us to work on a coarse to fine estimation of artificial image sequences that shows good convergence properties. It is also applied to natural image sequences.

Motion-Based Compression of Underwater Video Imagery for the Operations of Unmanned Submersible Vehicles

Unmanned vehicles are employed more and more frequently for a range of scientific and commercial undersea applications. However, the critical dependency on a tether link, mainly for the transmission of live images to the surface for command and control, is a significant technological obstacle limiting vehicle maneuverability. The elimination of the tether requires the capability to compress a massive amount of live video data to meet the bandwidth limitations of acoustic telemetry. This paper addresses motion-compensated compression of underwater video imagery and the extraction of the sought-after transformations by the application of the brightness constancy assumption and a generalized dynamic image model for the analysis of time-varying imagery. Two approaches, suitable for automatic vision-based navigation or operator-assisted missions of unmanned submersible vehicles, are considered. In the former case, the 3D motion and position information extracted from the raw data by the vision-based navigation system is used directly to perform the compression. In the latter case, the compression is achieved using the motion and radiometric information extracted from the live and reconstructed images at the surface station. To evaluate the performance of proposed methods, results from experiments with synthetic and real data are presented and compared to compression methods that are based on the brightness constancy model.