Energy Minimization Formulation Research Articles

Joint Stereo Video Deblurring, Scene Flow Estimation and Moving Object Segmentation.

Stereo videos for the dynamic scenes often show unpleasant blurred effects due to the camera motion and the multiple moving objects with large depth variations. Given consecutive blurred stereo video frames, we aim to recover the latent clean images, estimate the 3D scene flow and segment the multiple moving objects. These three tasks have been previously addressed separately, which fail to exploit the internal connections among these tasks and cannot achieve optimality. In this paper, we propose to jointly solve these three tasks in a unified framework by exploiting their intrinsic connections. To this end, we represent the dynamic scenes with the piece-wise planar model, which exploits the local structure of the scene and expresses various dynamic scenes. Under our model, these three tasks are naturally connected and expressed as the parameter estimation of 3D scene structure and camera motion (structure and motion for the dynamic scenes). By exploiting the blur model constraint, the moving objects and the 3D scene structure, we reach an energy minimization formulation for joint deblurring, scene flow and segmentation. We evaluate our approach extensively on both synthetic datasets and publicly available real datasets with fast-moving objects, camera motion, uncontrolled lighting conditions and shadows. Experimental results demonstrate that our method can achieve significant improvement in stereo video deblurring, scene flow estimation and moving object segmentation, over state-of-the-art methods.

Slanted Stixels: A Way to Represent Steep Streets

This work presents and evaluates a novel compact scene representation based on Stixels that infers geometric and semantic information. Our approach overcomes the previous rather restrictive geometric assumptions for Stixels by introducing a novel depth model to account for non-flat roads and slanted objects. Both semantic and depth cues are used jointly to infer the scene representation in a sound global energy minimization formulation. Furthermore, a novel approximation scheme is introduced in order to significantly reduce the computational complexity of the Stixel algorithm, and then achieve real-time computation capabilities. The idea is to first perform an over-segmentation of the image, discarding the unlikely Stixel cuts, and apply the algorithm only on the remaining Stixel cuts. This work presents a novel over-segmentation strategy based on a fully convolutional network, which outperforms an approach based on using local extrema of the disparity map. We evaluate the proposed methods in terms of semantic and geometric accuracy as well as run-time on four publicly available benchmark datasets. Our approach maintains accuracy on flat road scene datasets while improving substantially on a novel non-flat road dataset.

Higher-Order Conditional Random Fields-Based 3D Semantic Labeling of Airborne Laser-Scanning Point Clouds

This paper presents a novel framework to achieve 3D semantic labeling of objects (e.g., trees, buildings, and vehicles) from airborne laser-scanning point clouds. To this end, we propose a framework which consists of hierarchical clustering and higher-order conditional random fields (CRF) labeling. In the hierarchical clustering, the raw point clouds are over-segmented into a set of fine-grained clusters by integrating the point density clustering and the classic K-means clustering algorithm, followed by the proposed probability density clustering algorithm. Through this process, we not only obtain a more uniform size and more homogeneous clusters with semantic consistency, but the topological relationships of the cluster’s neighborhood are implicitly maintained by turning the problem of topology maintenance into a clustering problem based on the proposed probability density clustering algorithm. Subsequently, the fine-grained clusters and their topological context are fed into the CRF labeling step, from which the fine-grained cluster’s semantic labels are learned and determined by solving a multi-label energy minimization formulation, which simultaneously considers the unary, pairwise, and higher-order potentials. Our experiments of classifying urban and residential scenes demonstrate that the proposed approach reaches 88.5% and 86.1% of “m F 1 ” estimated by averaging all classes of the F 1 -scores. We prove that the proposed method outperforms five other state-of-the-art methods. In addition, we demonstrate the effectiveness of the proposed energy terms by using an “ablation study” strategy.

A continuation method for rigid‐cohesive fracture in a discontinuous Galerkin finite element setting

SummaryAn energy minimization formulation of initially rigid cohesive fracture is introduced within a discontinuous Galerkin finite element setting with Nitsche flux. The finite element discretization is directly applied to an energy functional, whose term representing the energy stored in the interfaces is nondifferentiable at the origin. Unlike finite element implementations of extrinsic cohesive models that do not operate directly on the energy potential, activation of interfaces happens automatically when a certain level of stress encoded in the interface potential is reached. Thus, numerical issues associated with an external activation criterion observed in the previous literature are effectively avoided. Use of the Nitsche flux avoids the introduction of Lagrange multipliers as additional unknowns. Implicit time stepping is performed using the Newmark scheme, for which a dynamic potential is developed to properly incorporate momentum. A continuation strategy is employed for the treatment of nondifferentiability and the resulting sequence of smooth nonconvex problems is solved using the trust region minimization algorithm. Robustness of the proposed method and its capabilities in modeling quasistatic and dynamic problems are shown through several numerical examples.

Sparsity Within and Across Overlapping Groups

Recently, penalties promoting signals that are sparse within and across groups have been proposed. In this letter, we propose a generalization that allows to encode more intricate dependencies within groups. However, this complicates the realization of the threshold function associated with the penalty, which hinders the use of the penalty in energy minimization. We discuss how to sidestep this problem, and demonstrate the use of the modified penalty in an energy minimization formulation for an inverse problem.

Video Object Discovery and Co-Segmentation with Extremely Weak Supervision.

We present a spatio-temporal energy minimization formulation for simultaneous video object discovery and co-segmentation across multiple videos containing irrelevant frames. Our approach overcomes a limitation that most existing video co-segmentation methods possess, i.e., they perform poorly when dealing with practical videos in which the target objects are not present in many frames. Our formulation incorporates a spatio-temporal auto-context model, which is combined with appearance modeling for superpixel labeling. The superpixel-level labels are propagated to the frame level through a multiple instance boosting algorithm with spatial reasoning, based on which frames containing the target object are identified. Our method only needs to be bootstrapped with the frame-level labels for a few video frames (e.g., usually 1 to 3) to indicate if they contain the target objects or not. Extensive experiments on four datasets validate the efficacy of our proposed method: 1) object segmentation from a single video on the SegTrack dataset, 2) object co-segmentation from multiple videos on a video co-segmentation dataset, and 3) joint object discovery and co-segmentation from multiple videos containing irrelevant frames on the MOViCS dataset and XJTU-Stevens, a new dataset that we introduce in this paper. The proposed method compares favorably with the state-of-the-art in all of these experiments.

Piecewise‐planar Reconstruction of Multi‐room Interiors with Arbitrary Wall Arrangements

AbstractReconstructing the as‐built architectural shape of building interiors has emerged in recent years as an important and challenging research problem. An effective approach must be able to faithfully capture the architectural structures and separate permanent components from clutter (e.g. furniture), while at the same time dealing with defects in the input data. For many applications, higher‐level information on the environment is also required, in particular the shape of individual rooms. To solve this ill‐posed problem, state‐of‐the‐art methods assume constrained input environments with a 2.5D or, more restrictively, a Manhattan‐world structure, which significantly restricts their applicability in real‐world settings. We present a novel pipeline that allows to reconstruct general 3D interior architectures, significantly increasing the range of real‐world architectures that can be reconstructed and labeled by any interior reconstruction method to date. Our method finds candidate permanent components by reasoning on a graph‐based scene representation, then uses them to build a 3D linear cell complex that is partitioned into separate rooms through a multi‐label energy minimization formulation. We demonstrate the effectiveness of our method by applying it to a variety of real‐world and synthetic datasets and by comparing it to more specialized state‐of‐the‐art approaches.

Parametric Regression on the Grassmannian.

We address the problem of fitting parametric curves on the Grassmann manifold for the purpose of intrinsic parametric regression. We start from the energy minimization formulation of linear least-squares in Euclidean space and generalize this concept to general nonflat Riemannian manifolds, following an optimal-control point of view. We then specialize this idea to the Grassmann manifold and demonstrate that it yields a simple, extensible and easy-to-implement solution to the parametric regression problem. In fact, it allows us to extend the basic geodesic model to (1) a "time-warped" variant and (2) cubic splines. We demonstrate the utility of the proposed solution on different vision problems, such as shape regression as a function of age, traffic-speed estimation and crowd-counting from surveillance video clips. Most notably, these problems can be conveniently solved within the same framework without any specifically-tailored steps along the processing pipeline.

Dual Smoothing for Marine Oil Spill Segmentation

We present a novel marine oil spill segmentation method that characterizes two smoothing modules at the label level and the pixel level separately. At the label level, we exploit the rolling guidance filter for smoothing the label cost volumes. It enables scale-aware labeling and thus alleviates the ambiguous segmentation that blurs the detailed structures of oil spills. At the pixel level, we adapt a cooperative model for smoothing higher order pixel variations, which has the potential of preserving elongated strips that often arise in oil spills. We integrate the two smoothing modules operating at different levels into an energy minimization formulation, which is referred to as dual smoothing. The coupling of the two smoothing modules enables an effective complement to each other such that the specific structures of oil spills are accurately characterized. We compute the optimal labeling of the dual-smoothing framework based on graph cuts. The proposed dual-smoothing framework is especially effective in segmenting elongated and detailed oil spills, and the experimental results demonstrate its advantages over thresholding- and graph-cut-based segmentations.

PISA: pixelwise image saliency by aggregating complementary appearance contrast measures with edge-preserving coherence.

Driven by recent vision and graphics applications such as image segmentation and object recognition, computing pixel-accurate saliency values to uniformly highlight foreground objects becomes increasingly important. In this paper, we propose a unified framework called pixelwise image saliency aggregating (PISA) various bottom-up cues and priors. It generates spatially coherent yet detail-preserving, pixel-accurate, and fine-grained saliency, and overcomes the limitations of previous methods, which use homogeneous superpixel based and color only treatment. PISA aggregates multiple saliency cues in a global context, such as complementary color and structure contrast measures, with their spatial priors in the image domain. The saliency confidence is further jointly modeled with a neighborhood consistence constraint into an energy minimization formulation, in which each pixel will be evaluated with multiple hypothetical saliency levels. Instead of using global discrete optimization methods, we employ the cost-volume filtering technique to solve our formulation, assigning the saliency levels smoothly while preserving the edge-aware structure details. In addition, a faster version of PISA is developed using a gradient-driven image subsampling strategy to greatly improve the runtime efficiency while keeping comparable detection accuracy. Extensive experiments on a number of public data sets suggest that PISA convincingly outperforms other state-of-the-art approaches. In addition, with this work, we also create a new data set containing 800 commodity images for evaluating saliency detection.

Fast and globally convex multiphase active contours for brain MRI segmentation

Multiphase active contour based models are useful in identifying multiple regions with spatial consistency but varying characteristics such as the mean intensities of regions. Segmenting brain magnetic resonance images (MRIs) using a multiphase approach is useful to differentiate white and gray matter tissue for anatomical, functional and disease studies. Multiphase active contour methods are superior to other approaches due to their topological flexibility, accurate boundaries, robustness to image variations and adaptive energy functionals. Globally convex methods are furthermore initialization independent. We extend the relaxed globally convex Chan and Vese two-phase piecewise constant energy minimization formulation of Chan et al. (2006) [1] to the multiphase domain and prove the existence of a global minimizer in a specific space which is one of the novel contributions of the paper. An efficient dual minimization implementation of our binary partitioning function model accurately describes disjoint regions using stable segmentations by avoiding local minima solutions. Experimental results indicate that the proposed approach provides consistently better accuracy than other related multiphase active contour algorithms using four different error metrics (Dice, Rand Index, Global Consistency Error and Variation of Information) even under severe noise, intensity inhomogeneities, and partial volume effects in MRI imagery.

Reconstruction of coronary trees from 3DRA using a 3D+t statistical cardiac prior.

A 3D+t description of the coronary tree is important for diagnosis of coronary artery disease and therapy planning. In this paper, we propose a method for finding 3D+t points on coronary artery tree given tracked 2D+t point locations in X-ray rotational angiography images. In order to cope with the ill-posedness of the problem, we use a bilinear model of ventricle as a spatio-temporal constraint on the nonrigid structure of the coronary artery. Based on an energy minimization formulation, we estimate i) bilinear model parameters, ii) global rigid transformation between model and X-ray coordinate systems, and iii) correspondences between 2D coronary artery points on X-ray images and 3D points on bilinear model. We validated the algorithm using a software coronary artery phantom.

Learning Component-Level Sparse Representation for Image and Video Categorization

A novel component-level dictionary learning framework that exploits image/video group characteristics based on sparse representation is introduced in this paper. Unlike the previous methods that select the dictionaries to best reconstruct the data, we present an energy minimization formulation that jointly optimizes the learning of both sparse dictionary and component-level importance within one unified framework to provide a discriminative and sparse representation for image/video groups. The importance measures how well each feature component represents the group property with the dictionary. Then, the dictionary is updated iteratively to reduce the influence of unimportant components, thus refining the sparse representation for each group. In the end, by keeping the top K important components, a compact representation is obtained for the sparse coding dictionary. Experimental results on several public image and video data sets are shown to demonstrate the superior performance of the proposed algorithm compared with the-state-of-the-art methods.

High Performance Imaging through Occlusion via Energy Minimization-Based Optimal Camera Selection

Seeing an object in a cluttered scene with severe occlusion is a significantly challenging task for many computer vision applications. Although camera array synthetic aperture imaging has proven to be an effective way for occluded object imaging, its imaging quality is often significantly decreased by the shadows of the foreground occluder. To overcome this problem, some recent research has been presented to label the foreground occluder via object segmentation or 3D reconstruction. However, these methods usually fail in the case of complicated occluder or severe occlusion. In this paper, we present a novel optimal camera selection algorithm to handle the problem above. Firstly, in contrast to the traditional synthetic aperture photography methods, we formulate the occluded object imaging as a problem of visible light ray selection from the optimal camera view. To the best of our knowledge, this is the first time to “mosaic” a high quality occluded object image via selecting multi-view optimal visible light rays from a camera array or a single moving camera. Secondly, a greedy optimization framework is presented to propagate the visibility information among various depth focus planes. Thirdly, a multiple label energy minimization formulation is designed in each plane to select the optimal camera view. The energy is estimated in the 3D synthetic aperture image volume and integrates the multiple view intensity consistency, previous visibility property and camera view smoothness, which is minimized via graph cuts. Finally, we compare this approach with the traditional synthetic aperture imaging algorithms on UCSD light field datasets and our own datasets captured in indoor and outdoor environment, and extensive experimental results demonstrate the effectiveness and superiority of our approach.

Smart Scribbles for Sketch Segmentation

AbstractWe present ‘Smart Scribbles’—a new scribble‐based interface for user‐guided segmentation of digital sketchy drawings. In contrast to previous approaches based on simple selection strategies, Smart Scribbles exploits richer geometric and temporal information, resulting in a more intuitive segmentation interface. We introduce a novel energy minimization formulation in which both geometric and temporal information from digital input devices is used to define stroke‐to‐stroke and scribble‐to‐stroke relationships. Although the minimization of this energy is, in general, an NP‐hard problem, we use a simple heuristic that leads to a good approximation and permits an interactive system able to produce accurate labellings even for cluttered sketchy drawings. We demonstrate the power of our technique in several practical scenarios such as sketch editing, as‐rigid‐as‐possible deformation and registration, and on‐the‐fly labelling based on pre‐classified guidelines.

A novel grain cluster-based homogenization scheme

An efficient homogenization scheme, termed the relaxed grain cluster (RGC), for elasto-plastic deformations of polycrystals is presented. The scheme is based on a generalization of the grain cluster concept. A volume element consisting of eight (= 2 × 2 × 2) hexahedral grains is considered. The kinematics of the RGC scheme is formulated within a finite deformation framework, where the relaxation of the local deformation gradient of each individual grain is connected to the overall deformation gradient by the, so-called, interface relaxation vectors. The set of relaxation vectors is determined by the minimization of the constitutive energy (or work) density of the overall cluster. An additional energy density associated with the mismatch at the grain boundaries due to relaxations is incorporated as a penalty term into the energy minimization formulation. Effectively, this penalty term represents the kinematical condition of deformation compatibility at the grain boundaries.Simulations have been performed for a dual-phase grain cluster loaded in uniaxial tension. The results of the simulations are presented and discussed in terms of the effective stress–strain response and the overall deformation anisotropy as functions of the penalty energy parameters. In addition, the prediction of the RGC scheme is compared with predictions using other averaging schemes, as well as to the result of direct finite element (FE) simulation. The comparison indicates that the present RGC scheme is able to approximate FE simulation results of relatively fine discretization at about three orders of magnitude lower computational cost.

Rigorous treatment of electrostatics for spatially varying dielectrics based on energy minimization

An energy minimization formulation of electrostatics that allows computation of the electrostatic energy and forces to any desired accuracy in a system with arbitrary dielectric properties is presented. An integral equation for the scalar charge density is derived from an energy functional of the polarization vector field. This energy functional represents the true energy of the system even in nonequilibrium states. Arbitrary accuracy is achieved by solving the integral equation for the charge density via a series expansion in terms of the equation's kernel, which depends only on the geometry of the dielectrics. The streamlined formalism operates with volume charge distributions only, not resorting to introducing surface charges by hand. Therefore, it can be applied to any spatial variation of the dielectric susceptibility, which is of particular importance in applications to biomolecular systems. The simplicity of application of the formalism to real problems is shown with analytical and numerical examples.

Shape from shading using graph cuts

In this paper, we investigate the applicability of graph cuts to the SFS (shape-from-shading) problem. We propose a new semi-global method for SFS using graph cuts. The new algorithm combines the local method proposed by Lee and Rosenfeld [C.H. Lee, A. Rosenfeld, Improved methods of estimating shape from shading using the light source coordinate system, Artif. Intell. 26 (1985) 125–143] and a global method using an energy minimization technique. By employing a new global energy minimization formulation, the convex/concave ambiguity problem of Lee and Rosenfeld's method can be resolved efficiently. A new combinatorial optimization technique, the graph cuts method, is used for the minimization of the proposed energy functional. Experimental results on a variety of synthetic and real-world images show that the proposed algorithm reconstructs the 3-D shape of objects very efficiently.

Estimation of optimal PDE-based denoising in the SNR sense

This paper is concerned with finding the best partial differential equation-based denoising process, out of a set of possible ones. We focus on either finding the proper weight of the fidelity term in the energy minimization formulation or on determining the optimal stopping time of a nonlinear diffusion process. A necessary condition for achieving maximal SNR is stated, based on the covariance of the noise and the residual part. We provide two practical alternatives for estimating this condition by observing that the filtering of the image and the noise can be approximated by a decoupling technique, with respect to the weight or time parameters. Our automatic algorithm obtains quite accurate results on a variety of synthetic and natural images, including piecewise smooth and textured ones. We assume that the statistics of the noise were previously estimated. No a priori knowledge regarding the characteristics of the clean image is required. A theoretical analysis is carried out, where several SNR performance bounds are established for the optimal strategy and for a widely used method, wherein the variance of the residual part equals the variance of the noise.

A new active contour model for shape extraction

We propose a new active contour model for shape extraction of objects in grey-valued two-dimensional images based on an energy-minimization formulation. The energy functional that we consider takes into account the two requirements of object isolation and smoothness of the contour. After deriving the Euler–Lagrange equations corresponding to the energy functional, we bring out some important geometric properties of a solution to these equations. The discussion on our solution method—with the help of which we try to minimize the energy functional by evolving an initial curve—also focuses on how to prescribe the initial curve fully automatically. The effectiveness of our algorithms is demonstrated with the help of experimental results. Copyright © 2000 John Wiley & Sons, Ltd.