Optical Flow Constraint Research Articles

Exploring the Temporal Consistency of Arbitrary Style Transfer: A Channelwise Perspective.

Arbitrary image stylization by neural networks has become a popular topic, and video stylization is attracting more attention as an extension of image stylization. However, when image stylization methods are applied to videos, unsatisfactory results that suffer from severe flickering effects appear. In this article, we conducted a detailed and comprehensive analysis of the cause of such flickering effects. Systematic comparisons among typical neural style transfer approaches show that the feature migration modules for state-of-the-art (SOTA) learning systems are ill-conditioned and could lead to a channelwise misalignment between the input content representations and the generated frames. Unlike traditional methods that relieve the misalignment via additional optical flow constraints or regularization modules, we focus on keeping the temporal consistency by aligning each output frame with the input frame. To this end, we propose a simple yet efficient multichannel correlation network (MCCNet), to ensure that output frames are directly aligned with inputs in the hidden feature space while maintaining the desired style patterns. An inner channel similarity loss is adopted to eliminate side effects caused by the absence of nonlinear operations such as softmax for strict alignment. Furthermore, to improve the performance of MCCNet under complex light conditions, we introduce an illumination loss during training. Qualitative and quantitative evaluations demonstrate that MCCNet performs well in arbitrary video and image style transfer tasks. Code is available at https://github.com/kongxiuxiu/MCCNetV2.

Exposing the Self-Supervised Space-Time Correspondence Learning via Graph Kernels

Self-supervised space-time correspondence learning is emerging as a promising way of leveraging unlabeled video. Currently, most methods adapt contrastive learning with mining negative samples or reconstruction adapted from the image domain, which requires dense affinity across multiple frames or optical flow constraints. Moreover, video correspondence predictive models require mining more inherent properties in videos, such as structural information. In this work, we propose the VideoHiGraph, a space-time correspondence framework based on a learnable graph kernel. Concerning the video as the spatial-temporal graph, the learning objectives of VideoHiGraph are emanated in a self-supervised manner for predicting unobserved hidden graphs via graph kernel manner. We learn a representation of the temporal coherence across frames in which pairwise similarity defines the structured hidden graph, such that a biased random walk graph kernel along the sub-graph can predict long-range correspondence. Then, we learn a refined representation across frames on the node-level via a dense graph kernel. The self-supervision of the model training is formed by the structural and temporal consistency of the graph. VideoHiGraph achieves superior performance and demonstrates its robustness across the benchmark of label propagation tasks involving objects, semantic parts, keypoints, and instances. Our algorithm implementations have been made publicly available at https://github.com/zyqin19/VideoHiGraph.

Reformulating Graph Kernels for Self-Supervised Space-Time Correspondence Learning.

Self-supervised space-time correspondence learning utilizing unlabeled videos holds great potential in computer vision. Most existing methods rely on contrastive learning with mining negative samples or adapting reconstruction from the image domain, which requires dense affinity across multiple frames or optical flow constraints. Moreover, video correspondence prediction models need to uncover more inherent properties of the video, such as structural information. In this work, we propose HiGraph+, a sophisticated space-time correspondence framework based on learnable graph kernels. By treating videos as a spatial-temporal graph, the learning objective of HiGraph+ is issued in a self-supervised manner, predicting the unobserved hidden graph via graph kernel methods. First, we learn the structural consistency of sub-graphs in graph-level correspondence learning. Furthermore, we introduce a spatio-temporal hidden graph loss through contrastive learning that facilitates learning temporal coherence across frames of sub-graphs and spatial diversity within the same frame. Therefore, we can predict long-term correspondences and drive the hidden graph to acquire distinct local structural representations. Then, we learn a refined representation across frames on the node-level via a dense graph kernel. The structural and temporal consistency of the graph forms the self-supervision of model training. HiGraph+ achieves excellent performance and demonstrates robustness in benchmark tests involving object, semantic part, keypoint, and instance labeling propagation tasks. Our algorithm implementations have been made publicly available at https://github.com/zyqin19/HiGraph.

Focusing dynamic single‐channel synthetic aperture radar video with optical flow‐informed reconstruction

The motion of targets is well known to result in their defocusing and displacement in SAR imagery, but detection of motion and re-focusing of targets under generic conditions remains an ongoing interest. One class of methods involves forming images of sub-apertures in which motion defocusing will be reduced. In this paper, dynamic tomographic image formation methods utilising an optical flow constraint is used to form a video of SAR sub-aperture images. These retain fine resolution of the full aperture, focusing along-track motion.

A dense light field reconstruction algorithm for four-dimensional optical flow constraint equation

Dense light field sampling is an important basis for refocusing, depth estimation and 3-D imaging. It is difficult to obtain high resolution dense light field with a large-scale camera array and expensive equipment. At the same time, the current storage devices and transmission bandwidth also limit this technology's post-processing and application. In order to effectively reconstruct the angle domain of the light field based on the sparse light field data, this paper analyzes the correlation and constraint relationship between the optical flow field and the motion field of multi-view images in the same scene, extends the traditional optical flow constraint equation of two-dimensional imaging to the optical flow constraint equation of four-dimensional light field, and establishes an effective mathematical model. The coordinate position of the original pixel in the new angle image is determined by coordinate search, and its intensity is reconstructed. The experimental results of multi-scene dense reconstruction show that the proposed method can reconstruct the texture, shadow and color information in the light field of a long-baseline scene with high quality. The quantitative evaluation results show that the algorithm can be applied to dense light field reconstruction of complex scenes. The algorithm in this paper is only suitable for the case of optical flow constraint in a linear light field, and the follow-up research will focus on the case of nonlinear optical flow constraint.

Structurally-constrained optical-flow-guided adversarial generation of synthetic CT for MR-only radiotherapy treatment planning

The rapid progress in image-to-image translation methods using deep neural networks has led to advancements in the generation of synthetic CT (sCT) in MR-only radiotherapy workflow. Replacement of CT with MR reduces unnecessary radiation exposure, financial cost and enables more accurate delineation of organs at risk. Previous generative adversarial networks (GANs) have been oriented towards MR to sCT generation. In this work, we have implemented multiple augmented cycle consistent GANs. The augmentation involves structural information constraint (StructCGAN), optical flow consistency constraint (FlowCGAN) and the combination of both the conditions (SFCGAN). The networks were trained and tested on a publicly available Gold Atlas project dataset, consisting of T2-weighted MR and CT volumes of 19 subjects from 3 different sites. The network was tested on 8 volumes acquired from the third site with a different scanner to assess the generalizability of the network on multicenter data. The results indicate that all the networks are robust to scanner variations. The best model, SFCGAN achieved an average ME of 0.9 5.9 HU, an average MAE of 40.4 4.7 HU and 57.2 1.4 dB PSNR outperforming previous research works. Moreover, the optical flow constraint between consecutive frames preserves the consistency across all views compared to 2D image-to-image translation methods. SFCGAN exploits the features of both StructCGAN and FlowCGAN by delivering structurally robust and 3D consistent sCT images. The research work serves as a benchmark for further research in MR-only radiotherapy.

High Simulation Reconstruction of Crowd Animation Based on Optical Flow Constraint Equation

Abstract The amount of information available in modern life is very high. And people's knowledge about what kind of genre they like or prefer, the richness of content and the frequency of use will affect the quality of their animation works and their style. Therefore, how to make animation reach a higher level of simulation has become one of the topics of social concern nowadays; with the advent of mobile Internet era, more and more people like this social platform, and in this environment, we need to personalize ourselves and integrate into it faster through various technical means. In this paper, researchers focus on the reconstruction of crowd movement trajectory based on the optical flow constraint equation (SDH) after modification. Since lasers, cameras and other devices have high-intensity pixel resolution characteristics as well as better directional characteristics, we can use these features to design and build the function; then we can complete the relevant actions according to the prototype model under certain conditions, realize the simulation of the scene according to its requirements, and finally complete the required functions of the whole system. Finally, the feasibility and effectiveness of the optical flow constraint equation and the optimized action model in practical applications are verified through experiments; at the same time, this also provides reference values for relevant staff.

Joint Motion Estimation and Compensation for Four-Dimensional Cone-Beam Computed Tomography Image Reconstruction

Four-Dimensional Cone-Beam Computed Tomography (4D-CBCT) imaging is important in upper abdomen tumor radiation treatment as it can obtain the tumor motion information immediately before treatment. However, the number of projections at a single phase for 4D-CBCT imaging is very small, and the CT images reconstructed by using conventional algorithms will be contaminated by view aliasing artifacts and noises. To address this problem, we propose a framework to jointly estimate and compensate the inter-phase motion for 4D-CBCT image reconstruction. Specifically, by introducing the intensity-based optical flow (OF) constraint into the reconstruction framework, the model can deal with the inter-frame displacements and improve the image quality simultaneously. The primal-dual algorithm method was used to optimize the cost function of the proposed model. Experiments on physical phantoms and patient data show that the proposed approach can effectively reduce the noise and artifacts in 4D–CBCT images and achieve the inter-phase motion vector fields (MVFs).

Motion Compensated Dynamic MRI Reconstruction With Local Affine Optical Flow Estimation.

This paper proposes a novel framework to reconstruct dynamic magnetic resonance imaging (DMRI) with motion compensation (MC). Specifically, by combining the intensity-based optical flow constraint with the traditional compressed sensing scheme, we are able to jointly reconstruct the DMRI sequences and estimate the interframe motion vectors. Then, the DMRI reconstruction can be refined through MC with the estimated motion field. By employing the coarse-to-fine multi-scale resolution strategy, we are able to update the motion field in different spatial scales. The estimated motion vectors need to be interpolated to the finest resolution scale to compensate the DMRI reconstruction. Moreover, the proposed framework is capable of handling a wide class of prior information (regularizations) for DMRI reconstruction, such as sparsity, low rank, and total variation. The formulated optimization problem is solved by a primal-dual algorithm with linesearch due to its efficiency when dealing with non-differentiable problems. Experiments on various DMRI datasets validate the reconstruction quality improvement using the proposed scheme in comparison to several state-of-the-art algorithms.

Technique for two-dimensional displacement field determination using a reliability-guided spatial-gradient-based digital image correlation algorithm.

This paper proposed a novel in-plane displacement field measurement algorithm using an optical flow strategy. We built a linear illumination model between images before and after deformation to guarantee intensity invariability. We used image upsampling and a reliability-guided strategy to find the matching points accurate to 0.5pixels in the reference and deformed images. The criterion to determine the reliability is zero-mean normalized cross-correlation coefficient. Afterward, we used the brightness constancy constraint combined with Lucas-Kanade optical flow constraint in a specific image region to obtain an overdetermined linear equation. We applied the noniterative least-squares algorithm to solve the equations and to achieve the displacement offsets. This research utilized multithread calculation to handle the complete cracking applications. We estimated the computing efficiency and calculation precision of the proposed method through a series of experimental speckle patterns. All results demonstrated the correctness, effectiveness, and robustness of the proposed method.

Visual servoing based on efficient histogram information

The robustness of a visual servoing task depends mainly on the efficiency of visual selections captured from a sensor at each robot’s position. A task function could be described as a regulation of the values sent via the control law to the camera velocities. In this paper we propose a new approach that does not depend on matching and tracking results. Thus, we replaced the classical minimization cost by a new function based on probability distributions and Bhattacharyya distance. To guarantee more robustness, the information related to the observed images was expressed using a combination of orientation selections. The new visual selections are computed by referring to the disposition of Histograms of Oriented Gradients (HOG) bins. For each bin we assign a random variable representing gradient vectors in a particular direction. The new entries will not be used to establish equations of visual motion but they will be directly inserted into the control loop. A new formulation of the interaction matrix has been presented according to the optical flow constraint and using an interpolation function which leads to a more efficient control behaviour and to more positioning accuracy. Experiments demonstrate the robustness of the proposed approach with respect to varying work space conditions.

Segmentation of the left ventricle in short‐axis sequences by combining deformation flow and optical flow

To help the clinicians to segment the borders of the left ventricle (LV) efficiently during measurement of the heart, the authors come up with a semi-automatic approach in this study that is capable of identifying the endocardial borders robustly from cine magnetic resonance images. Firstly, the deformation flow is computed between the inputted boundary in the previous frame and the extracted edge of the LV in the current frame based on boundary minimum distance principle (BMDP). Then, the deformation flow is constrained by optical flow calculated by a partial differential equation model. A smooth deformation boundary is then formed by minimising the energy between the previously inputted boundary and the rough boundary obtained by BMDP and optical flow constraint. To extract edge of the LV as accurate as possible, a threshold selection method is used and improved based on the previous study. The proposed approach is tested on the open access dataset. The computed average perpendicular distance is 1.36 ± 0.24 mm and the computed Dice measure is 90.7% ± 0.15%. Experimental results show that the proposed approach is significantly more accurate than the referenced state of art methods.

Direct hand pose estimation for immersive gestural interaction

This paper presents a novel approach for performing intuitive gesture-based interaction using depth data acquired by Kinect. The main challenge to enable immersive gestural interaction is dynamic gesture recognition. This problem can be formulated as a combination of two tasks; gesture recognition and gesture pose estimation. Incorporation of fast and robust pose estimation method would lessen the burden to a great extent. In this paper we propose a direct method for real-time hand pose estimation. Based on the range images, a new version of optical flow constraint equation is derived, which can be utilized to directly estimate 3D hand motion without any need of imposing other constraints. Extensive experiments illustrate that the proposed approach performs properly in real-time with high accuracy. As a proof of concept, we demonstrate the system performance in 3D object manipulation on two different setups; desktop computing, and mobile platform. This reveals the system capability to accommodate different interaction procedures. In addition, a user study is conducted to evaluate learnability, user experience and interaction quality in 3D gestural interaction in comparison to 2D touchscreen interaction.

Observation model based on scale interactions for optical flow estimation.

In this paper, an original observation model for multiresolution optical flow estimation is introduced. Multiresolution frameworks, often based on coarse-to-fine warping strategies, are widely used by state-of-the-art optical flow methods. They allow the recovery of large motions by successive estimations of the flow field at several resolution levels. Although such approaches perform very efficiently and usually lead to faster minimizations, they generally consider independent problems at each resolution levels and do not exploit the existing interactions between scales (especially the influences of fine scales on larger ones). In this paper, we tackle this issue by proposing a flexible framework, inspired from fluid mechanics, able to partly counter these limitations. For each resolution level, our process filters the equations of interest and decomposes the key variables into resolved (i.e., at a given resolution) and unresolved (i.e., at finer resolutions) components. This enables to derive a new data term that takes into account, at coarse resolutions, the influence of their unresolved parts. From this new term, we propose two different estimation strategies, depending on whether we explicitly know the type of relations between the different scales (as for physical processes) or not. In order to test the efficiency of this new observation model, we have embedded it in a simple multiresolution Lucas-Kanade estimator. Comparing the usual optical flow constraint equation with this new term in the same motion estimation procedure, it clearly appears that the proposed term leads to more consistent estimates and prevents from errors propagation apparition during the estimation. In all situations (synthetic, real, physical images or not), our new term is able to greatly improve the results compared with usual conservation constraints.

A Local Optical Flow Constraint Target Extracting Algorithm of Kalman Filter Based on Background Modeling

Owing to the problems of inter frame difference method cannot extract the entire target and most optical flow algorithms with computational time, poor real-time performance, a local optical flow constraint target extracting algorithm of Kalman filter based on background modeling is proposed. Firstly use Kalman filter method based on background modeling predict and update the background, then make Lucas-Kanade local optical flow algorithm search the background changing region, finally determine the gray contour, extract target. Compared with the classical algorithms, the simulation results show that new algorithm can extract the moving object quickly and accurately, and has better robustness to the environment changes and target circuitous movement.

Conditional simultaneous localization and mapping: A robust visual SLAM system

Abstract Visual simultaneous localization and mapping (VSLAM) is becoming increasingly popular in research and industry as a solution for mapping an unknown environment with moving cameras. However, classic methods such as the Extended Kalman Filter (EKF)-based VSLAM have two significant limitations: First, their robustness and accuracy drop dramatically when low frame rate cameras are used or sudden changes in camera velocity occur. Second, their dynamic models are expensive to build, or are too simple to simulate complex movements. In this paper, a novel VSLAM approach called conditional simultaneous localization and mapping (C-SLAM) is proposed in which camera state transition is derived from image data using optical flow constraints and epipolar geometry in the prediction stage. This improvement not only increases prediction accuracy but also replaces commonly used predefined dynamic models which require additional computation. Compared to classic VSLAM approaches, C-SLAM performs more accurately in prediction and has high computational efficiency, especially under conditions such as abrupt changes in camera velocity or low camera frame rate. Such advantages are supported by the experimental results and analysis presented in this paper.

一种快速的单目移动相机下运动目标检测方法

Based on projective geometry,the relation between the moving camera and still scene was analyzed,and a fast estimation algorithm was proposed based on the optical center coordinate system.On this coordinate in which the camera was always static,the scene and the targets were moving instead. The problem of moving target detection on a moving camera was transformed into distinguishing independent movement of targets from the global motion of scenes on a still platform and a concise form of optical flow constraint was deduced.This framework compensated the random movement according to the optical flow field which was got from KLT.Compared with the original algorithm,the computing speed increased about 10times while ensuring accuracy and robustness.Experiments were performed based on actual image data,and the results show that this model can reduce the computation time extreamly under the premise of ensuring the accuracy and robustness,and be pratical on moving target detection with a moving camera without construction.

A Fast Algorithm Research of Moving Targets Detection Based on GPU

Moving object detection which is an important part of the digital image processing technology is research focus and difficulty in computer vision, pattern recognition, object recognition and tracking, moving image coding, security monitoring and other fields. It also has a wide application prospect in the military, national defense and industry fields. However, when there are large amount of moving targets and exercise conditions, the amount of data generated is also inestimable and target detection is a bottleneck in the data processing at the same time. The work of this paper is to simplify motion detection in optical flow constraint equation solving process and all the solving processes were constructed on GPU for fast speed, then it forms a fast moving target detection.

Persistent tracking of static scene features using geometry

Despite many alternatives to feature tracking problem, iterative least squares solution solving the optical flow constraint has been the most popular approach used by many in the field. This paper attempts to leverage the former efforts to enhance feature tracking methods by introducing a view geometric constraint to the tracking problem. In contrast to alternative geometry based methods, the proposed approach provides a closed form solution to optical flow estimation from image appearance and view geometry constraints. We particularly use invariants in the projective coordinates generated from tracked features that results in a new optical flow equation. This treatment provides persistent tracking of features even when they are occluded. At the end of each tracking loop the quality of the tracked features is judged using both appearance similarity and geometric consistency. Our experiments demonstrate robust tracking performance even when the features are occluded or they undergo appearance changes due to projective deformation of the template.

Horn-Schunck Optical Flow with a Multi-Scale Strategy

The seminal work of Horn and Schunck is the first variational method for optical flow estimation. It introduced a novel framework where the optical flow is computed as the solution of a minimization problem. From the assumption that pixel intensities do not change over time, the optical flow constraint equation is derived. This equation relates the optical flow with the derivatives of the image. There are infinitely many vector fields that satisfy the optical flow constraint, thus the problem is ill-posed. To overcome this problem, Horn and Schunck introduced an additional regularity condition that restricts the possible solutions. Their method minimizes both the optical flow constraint and the magnitude of the variations of the flow field, producing smooth vector fields. One of the limitations of this method is that, typically, it can only estimate small motions. In the presence of large displacements, this method fails when the gradient of the image is not smooth enough. In this work, we describe an implementation of the original Horn and Schunck method and also introduce a multi-scale strategy in order to deal with larger displacements. For this multi-scale strategy, we create a pyramidal structure of downsampled images and change the optical flow constraint equation with a nonlinear formulation. In order to tackle this nonlinear formula, we linearize it and solve the method iteratively in each scale. In this sense, there are two common approaches: one approach that computes the motion increment in the iterations; or the one we follow, that computes the full flow during the iterations. The solutions are incrementally refined over the scales. This pyramidal structure is a standard tool in many optical flow methods.