Motion Boundaries Research Articles

Refined TV-L 1 Optical Flow Estimation Using Joint Filtering

Though the accuracy and robustness of optical flow has been dramatically enhanced over the past few years, the issue of edge-blurring near the image and motion boundaries has remained a challenge in flow field estimation. In this paper, we propose a refined total variation with L 1 norm (TV- L 1) optical flow estimation approach using joint filtering, named JOF. First, we divide the image into three categorized regions: mutual-structure regions, inconsistent structure regions, and smooth regions. The mutual-structure guided filter for optical flow estimation is constructed by extracting the mutual-structure regions of the flow field. Second, the refined TV- L 1 optical flow model is proposed by incorporating the non-local term and mutual-structure guided filter objective function into the classical TV- L 1 energy function. Furthermore, the novel TV- L 1 optical flow objective function is minimized using a joint filtering program composed of a weighted median filter and a mutual-structure guided filter to optimize the estimated flow field during the coarse-to-fine optical flow computation scheme. Finally, we compare the proposed JOF method with several state-of-the-art approaches including variational and deep learning based optical flow models using the Middlebury, MPI-Sintel, and UCF101 test databases. The evaluation results indicate that the proposed method has high accuracy and good robustness for flow field computation and, especially, the significant benefit of edge-preserving.

Three-Zone Segmentation Based Motion Compensation for Video Compression.

Motion compensation has been widely employed for removing temporal redundancies in typical hybrid video coding framework. The popular video compression standards, such as H.264/AVC and HEVC, adopt the block based partitioning model to describe the motion filed due to its high compression efficiency and relatively low computational complexity. However, block based motion compensation may not align with the actual object motion boundaries, potentially limiting the compression efficiency. In view of this, we propose a three-zone segmentation based motion compensation scheme to improve the description accuracy of motion field as well as the coding efficiency. In particular, the segmentation information is implied in the reference frame instead of being explicitly signalled. Based on the segmentation information, three motion compensation zones can be identified, including one edge, one foreground, and one background zone. The foreground zone is motion compensated by the signalled motion vector of the block, and the background zone is motion compensated by the motion information implicitly derived from the local motion field. Regarding the edge zone, it is viewed as an overlapped area and the weighted compensation strategy is adopted. The proposed algorithm is implemented into the reference software VTM-1.0 of Versatile Video Coding (VVC), and simulation results show that the algorithm can achieve 1.14% and 1.06% bitrate savings for random access and lowdelay configurations, respectively.

Video saliency detection based on low-level saliency fusion and saliency-aware geodesic

We present a spatiotemporal saliency detection method for videos. In contrast to previous methods that focus on exploiting the different underlying saliency cues or ignore motion information, the proposed method aims to use both appearance information based on spatial edges and spatial color saliency and motion information based on temporal motion boundaries as indicators of foreground object locations. Spatial color saliency is obtained from the fusion of three color features: color edge connectivity, color rarity, and color compactness. Subsequently, we further smooth the color saliency to eliminate background noises and to further boost the detection accuracy. Then, we propose a strategy-based low-level saliency fusion that guarantees to complementarily employ the smoothed color saliency, spatial edges, and temporal motion boundaries clues toward producing high-accuracy low-level saliency. Subsequently, we generate framewise spatiotemporal saliency maps using a geodesic distance from the low-level saliency. Subsequently, high-quality results are obtained through the geodesic distance to the background area in the subsequent frames. Extensive quantitative and qualitative experiments on three public video datasets demonstrate the superiority of the proposed method over the state-of-the-art algorithms.

SemFlow: Semantic-Driven Interpolation for Large Displacement Optical Flow

This paper presents a semantic-guided interpolation scheme (SemFlow) to handle motion boundaries and occlusions in large displacement optical flow. The basic idea is to segment images into superpixels and estimate their homographies for interpolation. In order to ensure each superpixel can be approximated as a plane, a semantic-guided refinement method is introduced. Moreover, we put forward a homography estimation model weighted by the distance between each superpixel and its K-nearest neighbors. Our newly-proposed distance metric combines the texture and semantic information to find proper neighbors. Our homography model performs better than the original affine model, since it accords with the real world projection relationship. The experiments on KITTI dataset demonstrate that SemFlow outperforms other state-of-the-art methods, especially in solving the problem of large scale motions and occlusions.

Discontinuity Preserving Liver MR Registration with 3D Active Contour Motion Segmentation.

The sliding motion of the liver during respiration violates the homogeneous motion smoothness assumption in conventional non-rigid image registration and commonly results in compromised registration accuracy. This paper presents a novel approach, registration with 3D active contour motion segmentation (RAMS), to improve registration accuracy with discontinuity-aware motion regularization. A Markov random field-based discrete optimization with dense displacement sampling and self-similarity context metric is used for registration, while a graph cuts-based 3D active contour approach is applied to segment the sliding interface. In the first registration pass, a mask-free L1 regularization on an image-derived minimum spanning tree is performed to allow motion discontinuity. Based on the motion field estimates, a coarse segmentation finds the motion boundaries. Next, based on MR signal intensity, a fine segmentation aligns the motion boundaries with anatomical boundaries. In the second registration pass, smoothness constraints across the segmented sliding interface are removed by masked regularization on a minimum spanning forest and masked interpolation of the motion field. For in vivo breath-hold abdominal MRI data, the motion masks calculated by RAMS are highly consistent with manual segmentations in terms of Dice similarity and bidirectional local distance measure. These automatically obtained masks are shown to substantially improve registration accuracy for both the proposed discrete registration as well as conventional continuous non-rigid algorithms. The presented results demonstrated the feasibility of automated segmentation of the respiratory sliding motion interface in liver MR images and the effectiveness of using the derived motion masks to preserve motion discontinuity.

Unsupervised Video Object Foreground Segmentation and Co-Localization by Combining Motion Boundaries and Actual Frame Edges

In this article the authors proposed a fast and fully unsupervised approach for a foreground object co-localization and segmentation of unconstrained videos. This article first computes both the actual edges and motion boundaries of the video frames, and then aligns them by the proposed HOG affinity map approach. Then, by filling the occlusions generated by the aligned edges, the paper obtained more precise masks about the foreground object. With an accumulation process, these masks could be derived as the motion-based likelihood, which is used as a unary term in the proposed graph model. Another unary term is called color-based likelihood, which is computed by the color distribution of foreground and background. Experiment results shows the method is fast and effective to detect and segment foreground objects.

Robust Optical Flow Estimation in Cardiac Ultrasound Images Using a Sparse Representation.

This paper introduces a robust 2-D cardiac motion estimation method. The problem is formulated as an energy minimization with an optical flow-based data fidelity term and two regularization terms imposing spatial smoothness and the sparsity of the motion field in an appropriate cardiac motion dictionary. Robustness to outliers, such as imaging artefacts and anatomical motion boundaries, is introduced using robust weighting functions for the data fidelity term as well as for the spatial and sparse regularizations. The motion fields and the weights are computed jointly using an iteratively re-weighted minimization strategy. The proposed robust approach is evaluated on synthetic data and realistic simulation sequences with available ground-truth by comparing the performance with state-of-the-art algorithms. Finally, the proposed method is validated using two sequences of in vivo images. The obtained results show the interest of the proposed approach for 2-D cardiac ultrasound imaging.

Reachable Relative Motion Design of Space Manipulator Actuated Microgravity Platform

This paper investigates the concept of the space manipulator-based microgravity platform. The system is composed of a spacecraft, a manipulator with an isolating shield as the end effector, and a testbed floating inside the shield. Assuming that the orbit of the spacecraft is circular, this paper focuses on the reachable relative motion design problem. Based on the analytical solution to the Clohessy–Wiltshire equation and the inverse kinematics of the manipulator, the relationship between the initial variables of the manipulator and the relative trajectory is presented. In both the zero-velocity releasing case and the nonzero-velocity releasing case, by investigating the boundaries of the relative motion, the criteria for the reachable relative motion are obtained. In the case in which the initial residual speed is adjustable, the routine solving for the largest elliptical trajectory is developed. Numerical simulations demonstrate the effectiveness of the criteria for the reachable motion in multiple cases.

Online background subtraction with freely moving cameras using different motion boundaries

We propose a method for online background subtraction from a successive-frame video captured using a freely moving camera. Our method exploits a technique of interactive image segmentation with seeds (the subsets of pixels marked as “foreground” and “background”). The key novelty of our method is to automatically estimate the seeds by exploiting two different motion boundaries that are respectively computed using the magnitude and direction of the flow field. The magnitude of flow field is likely to be useful in differentiating the foreground and background motions when the moving objects and the camera make a movement towards the same direction. In contrast, the direction of flow field helps in discriminating the observed motions when the amount of displacement of the moving objects and the camera is the same. By adaptively exploiting the advantages of these different motion boundaries, our method enables to estimate the reliable foreground/background seeds. With the estimated seeds, our method performs accurate background subtraction even when the complex camera movements (e.g., large pan-tilt-zoom, rotation) are made. Our experiments demonstrate the effectiveness of our method using public dataset and other real image sequences.

Multi-objective path planning for unmanned surface vehicle with currents effects

This paper investigates the path planning problem for unmanned surface vehicle (USV), wherein the goal is to find the shortest, smoothest, most economical and safest path in the presence of obstacles and currents, which is subject to the collision avoidance, motion boundaries and velocity constraints. We formulate this problem as a multi-objective nonlinear optimization problem with generous constraints. Then, we propose the dynamic augmented multi-objective particle swarm optimization algorithm to achieve the solution. With our approach, USV can select the ideal path from the Pareto optimal paths set. Numerical simulations verify the effectiveness of our formulated model and proposed algorithm.

Irreversible passive energy transfer of an immersed beam subjected to a sinusoidal flow via local nonlinear attachment

Passive vibration control of immersed structures in deep water subjected to the external fluid flow has rarely been studied. With the aim of passively absorbing, vibration of a doubly fixed beam under a sinusoidal flow using a nonlinear attachment is studied in this paper. The beam is modeled using the Euler-Bernoulli beam theory. The essentially nonlinear attachment has purely cubic stiffness and linear damping. Required conditions for occurring relaxation oscillations, Hopf and saddle-node bifurcations in the system response are studied. Analytical and numerical approaches are used to analyze the steady-state responses of the considered model. In addition, the influence of the location, the damping and the stiffness of the nonlinear absorber and external load are investigated on the dynamical behavior of the system. Furthermore, influence of the attachment stiffness on quasi-periodic motion boundaries surveyed. It is shown that in the presence of the nonlinear energy pumping the amplitude of the system response decreases significantly and as the attachment location approaches the beam supports the occurrence of the relaxation oscillations decreases and saddle-node bifurcations take place in the system response. The presented results and the proposed modeling approach can be used to design and enhance the performance of nonlinear vibration absorbers for subsea structures.

High-Resolution Mapping of Chromatin Dynamics during Transcription in Mammary Tumor Cells

Chromatin is defined to be highly dynamic for the regulation of gene expression. Control of gene expression mostly occurs at the transcription stage based on modifications of chromatin structure and transcription factors (estrogen-receptor alpha (ERα), in our case) along with RNA polymerase II dynamics. However, it remains unclear how structural chromatin complexes respond dynamically through coordinate movements to regulate genomic processes. Also, the global and local motion correlation of chromatin with other transcription players is still debated. For this purpose, we developed a method to estimate motion at sub-pixel resolution with high accuracy, which reveals sub-diffraction vectorial information, although the intensity is diffraction-limited, based on reconstructed dense global flow fields of fluorescent images. This approach allows us to quantify the biophysical properties such as the diffusion coefficient or the underlying types of motion of H2B tagged with GFP, and SiR-DNA as chromatin markers, and ERα tagged GFP in MCF-7 breast cancer cells. We measure the correlated flows of chromatin markers, and ERα, based on direction and magnitude of motion at different time intervals. To follow the dynamic response during transcription, we studied the correlated motion of chromatin in the absence and presence of 17beta-estradiol (E2) for (de) activation of transcription along with transcription inhibitors at initiation and intermediate transcriptional stages. Quantification of spatial smoothness shows that distinct motion boundaries over large time intervals, while for short time intervals smooth transitions in a small neighborhood dominate. We observe similar trends in chromatin dynamics upon pausing transcription at two different stages and long range correlations during transcription.

Online Localization and Prediction of Actions and Interactions.

This paper proposes a person-centric and online approach to the challenging problem of localization and prediction of actions and interactions in videos. Typically, localization or recognition is performed in an offline manner where all the frames in the video are processed together. This prevents timely localization and prediction of actions and interactions - an important consideration for many tasks including surveillance and human-machine interaction. In our approach, we estimate human poses at each frame and train discriminative appearance models using the superpixels inside the pose bounding boxes. Since the pose estimation per frame is inherently noisy, the conditional probability of pose hypotheses at current time-step (frame) is computed using pose estimations in the current frame and their consistency with poses in the previous frames. Next, both the superpixel and pose-based foreground likelihoods are used to infer the location of actors at each time through a Conditional Random Field enforcing spatio-temporal smoothness in color, optical flow, motion boundaries and edges among superpixels. The issue of visual drift is handled by updating the appearance models, and refining poses using motion smoothness on joint locations, in an online manner. For online prediction of action/interaction confidences, we propose an approach based on Structural SVM that operates on short video segments, and is trained with the objective that confidence of an action or interaction increases as time passes in a positive training clip. Lastly, we quantify the performance of both detection and prediction together, and analyze how the prediction accuracy varies as a time function of observed action/interaction at different levels of detection performance. Our experiments on several datasets suggest that despite using only a few frames to localize actions/interactions at each time instant, we are able to obtain competitive results to state-of-the-art offline methods.

Guided Filtering: Toward Edge-Preserving for Optical Flow

Despite progress made in the accuracy and robustness of optical flow in past years, the problem of over-segmentation and the blurring of image edge and motion boundary caused by the illumination change, complex texture, large displacement, and motion occlusion still remain. Recently, we developed a guided filtering scheme for flow field estimation, which is implemented as an add-on optimal operation during the coarse-to-fine optical flow computation. In this paper, we first review the research progress in optical flow computation and discuss limitations of the currently popular median filtering heuristic for a flow field optimization. We then introduce a general formulation of the guided filtering and provide the detailed illustration. Furthermore, we explore the potential of the guided filtering optimization for the flow field estimation under the coarse-to-fine computing scheme. Finally, we modify some typical and state-of-the-art optical flow methods by applying the proposed guided filtering operation to the baseline models, and test the performances of the basic and developed models through the Middlebury, MPI-Sintel, and KITTI data. The experimental results demonstrate that the guided filtering scheme is able to preserve the image edges and motion boundaries, and to improve the accuracy and robustness of optical flow estimation.

Scene Flow Estimation Based on 3D Local Rigidity Assumption and Depth Map Driven Anisotropic Smoothness

In this paper, we present a novel method to estimate the dense scene flow from the aligned depth map and color image using variational framework. For scene flow estimation, most scenes can be seen as scenes composed by independent 3-D rigid parts, so we apply 3-D local rigidity assumption to data term as a fidelity measure for each pixel. Meanwhile, in order to improve the accuracy of scene flow estimation at the boundaries of motion, we assume that depth map and color image are aligned and utilize the boundaries information of depth map to yield smoothness term which is weighted by a depth map driven anisotropic diffusion tensor. In addition, an efficient numerical algorithm named primal-dual algorithm is implemented for the variational formulation of scene flow estimation. Our method is tested on the Middlebury data sets, and the real-world scene data set captured by KINECT. Experimental results show that our method can receive dense and accurate scene flow and preserve motion boundaries well.

Human actions recognition: an approach based on stable motion boundary fields

Automatic video action recognition have been a long-standing problem in computer vision. To obtain a scalable solution for actions recognition, it is important to have efficient visual representation of motions. In this paper, we propose a new visual representation for actions based in the body motion boundaries. The first step, a set of optical flow frames highlighting the principal motions in the poses is substracted. Then, the motion boundaries are computed from the previous optical flow frames. Maximum Stable Extremal Regions are then applied to motion boundaries maps in order to obtain Motion Stable Shape (MSS) features. Local descriptors were computed based on each detected MSS to capture motion patterns. To predict the classes of the different human actions, we have represented different descriptors with a bag-of-words (BOW) model and for classification, we use a non-linear support vector machine. We have performed a set of experiments on different datasets: Weizmann, KTH, UCF sport, UCF50 and Hollywood to prove the efficiency of our developed model. The achieved results improve the state-of-the-art on the KTH and Weizmann datasets and are comparable to state-of-the-art for UCF sport and UCF50 datasets.

Better Dense Trajectories by Motion in Videos.

Currently, the most widely used point trajectories generation methods estimate the trajectories from the dense optical flow, by using a consistency check strategy to detect the occluded regions. However, these methods will miss some important trajectories, thus resulting in breaking smooth areas without any structure especially around the motion boundaries (MBs). We suggest exploring MBs in video to generate more accurate dense point trajectories. Estimating MBs from the video improves the point trajectory accuracy of the discontinuity or occluded areas. Then, we obtain trajectories by tracking the initial feature points through all frames. The experimental results demonstrate that our method outperforms the state-of-the-art methods on the challenging benchmark.

Spatiotemporal Colorization of Video Using 3D Steerable Pyramids

We propose a new technique for video colorization based on spatiotemporal color propagation in the 3D video volume, utilizing the dominant orientation response obtained from the steerable pyramid decomposition of the video. The volumetric color diffusion from the sources that are marked by scribbles occurs across spatiotemporally smooth regions, and the prevention of leakage is facilitated by the spatiotemporal discontinuities in the output of steerable filters, representing the object boundaries and motion boundaries. Unlike most existing methods, our approach dispenses with the need of motion vectors for interframe color transfer and provides a general framework for image and video colorization. The experimental results establish the effectiveness of the proposed approach in colorizing videos having different types of motion and visual content even in the presence of occlusion, in terms of accuracy and computational requirements.

Saliency flow based video segmentation via motion guided contour refinement

Video object segmentation is usually regarded as a pre-process for potential performance improvement in contemporary human activity studies. Given the fact that the moving target needs to be identified and localized before any sophisticated high-level analysis being subsequently carried out, recent segmentation researches choose to achieve high accuracy by modeling a spatiotemporal scheme using moving contour detection. Unfortunately, the obtained motion boundaries are prone to bring about unreliable objects’ localization due to the usage of solitude visual cues, which also motivated attemptation for more auxiliaries. Recently detection achievements have shown that saliency based high-level analysis is able to guide more reliable object localization, Thus, incorporating the saliency flow with other visual cues into an unified segmentation framework would be a promising way for more accurate video segmentation. In this paper, we formulate a unified segmentation framework based on a compositional model by combining saliency flow detection with motion estimation, and the flexibility of this framework also allows the other type of cues to be incorporated for further accuracy improvement. In addition, a robust contour filling method is proposed based on inter-frame edge prediction regardless whether the object contour is enclosed or not. With an efficient reliability evaluation and refinement of motion contours, the proposed segmentation strategy outperforms the other state-of-art approaches on two video object segmentation benchmark databases.

Fast Numerical Simulation of Focused Ultrasound Treatments During Respiratory Motion With Discontinuous Motion Boundaries.

Focused ultrasound (FUS) is rapidly gaining clinical acceptance for several target tissues in the human body. Yet, treating liver targets is not clinically applied due to a high complexity of the procedure (noninvasiveness, target motion, complex anatomy, blood cooling effects, shielding by ribs, and limited image-based monitoring). To reduce the complexity, numerical FUS simulations can be utilized for both treatment planning and execution. These use-cases demand highly accurate and computationally efficient simulations. We propose a numerical method for the simulation of abdominal FUS treatments during respiratory motion of the organs and target. Especially, a novel approach is proposed to simulate the heating during motion by solving Pennes' bioheat equation in a computational reference space, i.e.,the equation is mathematically transformed to the reference. The approach allows for motion discontinuities, e.g., the sliding of the liver along the abdominal wall. Implementing the solver completely on the graphics processing unit and combining it with an atlas-based ultrasound simulation approach yields a simulation performance faster than real time (less than 50-s computing time for 100s of treatment time) on a modern off-the-shelf laptop. The simulation method is incorporated into a treatment planning demonstration application that allows to simulate real patient cases including respiratory motion. The high performance of the presented simulation method opens the door to clinical applications. The methods bear the potential to enable the application of FUS for moving organs.