Motion Vector Field Research Articles

A Sports Training Video Classification Model Based on Deep Learning

A sports training video classification model based on deep learning is studied for targeting low classification accuracy caused by the randomness of objective movement in sports training video. The camera calibration technology is used to restore the position of the target in the real three-dimensional space. After the camera calibration in the video, the sports training video is preprocessed. The input video segment is divided into equal length segments to obtain the subvideo segment. The motion vector field, brightness feature, color feature, and texture feature of the subvideo segment are extracted, and the extracted features are input into the AlexNet convolutional neural network. ReLU is used as the activation function in this convolutional neural network. Local response normalization is used to suppress and enhance the output of neurons to highlight the performance of useful information, so that the output classification results are more accurate. Event matching method is used to match the convolutional neural network output to complete the sports training video classification. The experimental results of the proposed study show that the model can effectively solve the problems of target moving randomness. The classification accuracy of sports training video is more than 99%, and the classification speed is faster which is shown from the results of the experiments.

Deep learning-based coronary artery motion estimation and compensation for short-scan cardiac CT.

During a typical cardiac short scan, the heart can move several millimeters. As a result, the corresponding CT reconstructions may be corrupted by motion artifacts. Especially the assessment of small structures, such as the coronary arteries, is potentially impaired by the presence of these artifacts. In order to estimate and compensate for coronary artery motion, this manuscript proposes the deep partial angle-based motion compensation (Deep PAMoCo). The basic principle of the Deep PAMoCo relies on the concept of partial angle reconstructions (PARs), that is, it divides the short scan data into several consecutive angular segments and reconstructs them separately. Subsequently, the PARs are deformed according to a motion vector field (MVF) such that they represent the same motion state and summed up to obtain the final motion-compensated reconstruction. However, in contrast to prior work that is based on the same principle, the Deep PAMoCo estimates and applies the MVF via a deep neural network to increase the computational performance as well as the quality of the motion compensated reconstructions. Using simulated data, it could be demonstrated that the Deep PAMoCo is able to remove almost all motion artifacts independent of the contrast, the radius and the motion amplitude of the coronary artery. In any case, the average error of the CT values along the coronary artery is about 25HU while errors of up to 300HU can be observed if no correction is applied. Similar results were obtained for clinical cardiac CT scans where the Deep PAMoCo clearly outperforms state-of-the-art coronary artery motion compensation approaches in terms of processing time as well as accuracy. The Deep PAMoCo provides an efficient approach to increase the diagnostic value of cardiac CT scans even if they are highly corrupted by motion.

Phase2Phase: Respiratory Motion-Resolved Reconstruction of Free-Breathing Magnetic Resonance Imaging Using Deep Learning Without a Ground Truth for Improved Liver Imaging.

Respiratory binning of free-breathing magnetic resonance imaging data reduces motion blurring; however, it exacerbates noise and introduces severe artifacts due to undersampling. Deep neural networks can remove artifacts and noise but usually require high-quality ground truth images for training. This study aimed to develop a network that can be trained without this requirement. This retrospective study was conducted on 33 participants enrolled between November 2016 and June 2019. Free-breathing magnetic resonance imaging was performed using a radial acquisition. Self-navigation was used to bin the k-space data into 10 respiratory phases. To simulate short acquisitions, subsets of radial spokes were used in reconstructing images with multicoil nonuniform fast Fourier transform (MCNUFFT), compressed sensing (CS), and 2 deep learning methods: UNet3DPhase and Phase2Phase (P2P). UNet3DPhase was trained using a high-quality ground truth, whereas P2P was trained using noisy images with streaking artifacts. Two radiologists blinded to the reconstruction methods independently reviewed the sharpness, contrast, and artifact-freeness of the end-expiration images reconstructed from data collected at 16% of the Nyquist sampling rate. The generalized estimating equation method was used for statistical comparison. Motion vector fields were derived to examine the respiratory motion range of 4-dimensional images reconstructed using different methods. A total of 15 healthy participants and 18 patients with hepatic malignancy (50 ± 15 years, 6 women) were enrolled. Both reviewers found that the UNet3DPhase and P2P images had higher contrast (P < 0.01) and fewer artifacts (P < 0.01) than the CS images. The UNet3DPhase and P2P images were reported to be sharper than the CS images by 1 reviewer (P < 0.01) but not by the other reviewer (P = 0.22, P = 0.18). UNet3DPhase and P2P were similar in sharpness and contrast, whereas UNet3DPhase had fewer artifacts (P < 0.01). The motion vector lengths for the MCNUFFT800 and P2P800 images were comparable (10.5 ± 4.2 mm and 9.9 ± 4.0 mm, respectively), whereas both were significantly larger than CS2000 (7.0 ± 3.9 mm; P < 0.0001) and UNnet3DPhase800 (6.9 ± 3.2; P < 0.0001) images. Without a ground truth, P2P can reconstruct sharp, artifact-free, and high-contrast respiratory motion-resolved images from highly undersampled data. Unlike the CS and UNet3DPhase methods, P2P did not artificially reduce the respiratory motion range.

Side information hybrid generation based on improved motion vector field

The quality of side information has an important impact on the performance of a distributed video coding system. At present, the generation of side information is mainly based on the translational motion model using the inter-frame correlation for motion estimation. Considering that the generated side information is prone to block effect and ghosting, as well as the situation that the nonlinear motion is not fully considered and the intra-frame correlation is not fully utilized, a side information hybrid generation algorithm based on an improvement model for the motion vector field is proposed. The side information frame to be generated for the current Wyner-Ziv frame is divided into easy-to-estimate and difficult-to-estimate macroblocks. For difficult-to-estimate macroblocks, the Horn and Schunck dense optical flow method is used to generate reliable motion vectors, for easy-to-estimate macroblocks, the block matching method is used to generate unreliable motion vectors which are modified by the proposed scheme, and then, the improved motion vectors are used for motion compensation to produce the final side information frame. Experiment results show that the quality of side information obtained by using the improved motion vector field for motion compensation has been significantly improved, thus the overall performance of the distributed video coding system has been effectively improved.

Regression-Based Motion Vector Field for Video Coding

In this paper, we study a method for compensating the non-translational motion behavior in video coding. The proposed method models the motion field of a prediction block based on the motion information of the neighboring blocks by using a linear regression approach. In order to provide a finer granularity of motion vectors the Regression-based Motion Vector Field (RMVF) method derives the motion field in ${4}\times {4}$ sub-block accuracy. Such approach generates a smooth and more realistic motion vector field inside the prediction block. The motion field generated with RMVF is then used as a new merge mode along with other merge modes in VTM-2.0 test model of the Versatile Video Coding (H.266/VVC) standard. The conducted experiments with JVET CTC sequences illustrate that the proposed RMVF method provides 0.77%, 0.19% and 0.41% bitrate reductions with random access (RA), low delay B (LDB) and low delay P (LDP) configurations, respectively. Furthermore, this method provides on average 0.65% bitrate saving for the 360° sequences in equirectangular projection format (ERP) with RA configuration.

Interpolation Algorithm for Motion Vector Field of CTA Image and Accelerated Implementation Based on Uniform Grid and Multi-level B-spline

In order to solve the problems of slow operation speed and low registration accuracy of thin plate spline (TPS) interpolation method for motion vector field in coronary computed tomography angiography (CTA), a multi-level B-spline interpolation method (MBS) with uniform grid is proposed. On the one hand, the interpolation method used local B-spline to refine the sparse mesh layer by layer in a multiscale way to improve the accuracy of registration. On the other hand, it used the splitting matrix method to interpolate the motion vector field, greatly reducing the operation time of interpolation. The experimental results show that the proposed method can be used for CTA image registration efficiently.

Applying primary data from permanent stations for geodynamic zoning

Introduction. The article focuses on present-day geodynamic motion in order to carry out geodynamic zoning of territories. Geodynamic monitoring may be both regional, for instance, of the Russian Federation, Ural region, geological rock mass, and it may also be local, i.e. covering a deposit and enclosing rock mass. Permanent stations of Global Navigation Satellite System (GNSS) have been used as a source of data for deformation monitoring. Methodology included the method of visualizing geodynamic motions according to the results of cyclic geodetic measurements which makes it possible to single out active geological structures, blocks, and tectonic faults on reasonable grounds. Results. It has been shown that it is advisable to use not modules of observation station displacement vector values but their velocities reduced to an annual cycle as a key source of information on geodynamic motion at large spatial-temporal bases. It has been indicated that an important characteristic of geodynamic motion vector field is divergence which characterizes the degree of convergence or divergence of a vector flux. Summary. Basic theses have been identified of the method of present-day geodynamic motions monitoring and visualization in the form of a vector field according to the results of cyclic geodetic measurements. Based on experimental data, it has been determined that the present-day geodynamic motion is vortical being the indicator of active tectonic faulting.

Deformable motion compensation for interventional cone-beam CT

Image-guided therapies in the abdomen and pelvis are often hindered by motion artifacts in cone-beam CT (CBCT) arising from complex, non-periodic, deformable organ motion during long scan times (5–30 s). We propose a deformable image-based motion compensation method to address these challenges and improve CBCT guidance. Motion compensation is achieved by selecting a set of small regions of interest in the uncompensated image to minimize a cost function consisting of an autofocus objective and spatiotemporal regularization penalties. Motion trajectories are estimated using an iterative optimization algorithm (CMA-ES) and used to interpolate a 4D spatiotemporal motion vector field. The motion-compensated image is reconstructed using a modified filtered backprojection approach. Being image-based, the method does not require additional input besides the raw CBCT projection data and system geometry that are used for image reconstruction. Experimental studies investigated: (1) various autofocus objective functions, analyzed using a digital phantom with a range of sinusoidal motion magnitude (4, 8, 12, 16, 20 mm); (2) spatiotemporal regularization, studied using a CT dataset from The Cancer Imaging Archive with deformable sinusoidal motion of variable magnitude (10, 15, 20, 25 mm); and (3) performance in complex anatomy, evaluated in cadavers undergoing simple and complex motion imaged on a CBCT-capable mobile C-arm system (Cios Spin 3D, Siemens Healthineers, Forchheim, Germany). Gradient entropy was found to be the best autofocus objective for soft-tissue CBCT, increasing structural similarity (SSIM) by 42%–92% over the range of motion magnitudes investigated. The optimal temporal regularization strength was found to vary widely (0.5–5 mm−2) over the range of motion magnitudes investigated, whereas optimal spatial regularization strength was relatively constant (0.1). In cadaver studies, deformable motion compensation was shown to improve local SSIM by ∼17% for simple motion and ∼21% for complex motion and provided strong visual improvement of motion artifacts (reduction of blurring and streaks and improved visibility of soft-tissue edges). The studies demonstrate the robustness of deformable motion compensation to a range of motion magnitudes, frequencies, and other factors (e.g. truncation and scatter).

A Specific Algorithm Based on Motion Direction Prediction

In this paper, we study the estimation of motion direction prediction for fast motion and propose a threshold-based human target detection algorithm using motion vectors and other data as human target feature information. The motion vectors are partitioned into regions by normalization to form a motion vector field, which is then preprocessed, and then the human body target is detected through its motion vector region block-temporal correlation to detect the human body motion target. The experimental results show that the algorithm is effective in detecting human motion targets in videos with the camera relatively stationary. The algorithm predicts the human body position in the reference frame of the current frame in the video by forward mapping the motion vector of the current frame, then uses the motion vector direction angle histogram as a matching feature, and combines it with a region matching strategy to track the human body target in the predicted region, thus realizing the human body target tracking effect. The algorithm is experimentally proven to effectively track human motion targets in videos with relatively static backgrounds. To address the problem of sample diversity and lack of quantity in a multitarget tracking environment, a generative model based on the conditional variational self-encoder conditional generation of adversarial networks is proposed, and the performance of the generative model is verified using pedestrian reidentification and other datasets, and the experimental results show that the method can take advantage of the advantages of both models to improve the quality of the generated results.

Joint Motion Estimation and Compensation for Four-Dimensional Cone-Beam Computed Tomography Image Reconstruction

Four-Dimensional Cone-Beam Computed Tomography (4D-CBCT) imaging is important in upper abdomen tumor radiation treatment as it can obtain the tumor motion information immediately before treatment. However, the number of projections at a single phase for 4D-CBCT imaging is very small, and the CT images reconstructed by using conventional algorithms will be contaminated by view aliasing artifacts and noises. To address this problem, we propose a framework to jointly estimate and compensate the inter-phase motion for 4D-CBCT image reconstruction. Specifically, by introducing the intensity-based optical flow (OF) constraint into the reconstruction framework, the model can deal with the inter-frame displacements and improve the image quality simultaneously. The primal-dual algorithm method was used to optimize the cost function of the proposed model. Experiments on physical phantoms and patient data show that the proposed approach can effectively reduce the noise and artifacts in 4D–CBCT images and achieve the inter-phase motion vector fields (MVFs).

Motion‐Compensated Frame Interpolation Using Cellular Automata‐Based Motion Vector Smoothing

Motion‐Compensated Frame Interpolation (MCFI) is one of the common temporal‐domain tamper operations, and it is used to produce faked video frames for improving the visual qualities of video sequences. The instability of temporal symmetry results in many incorrect Motion Vectors (MVs) for Bidirectional Motion Estimation (BME) in MCFI. The existing Motion Vector Smoothing (MVS) works often oversmooth or revise correct MVs as wrong ones. To overcome this problem, we propose a Cellular Automata‐based MVS (CA‐MVS) algorithm to smooth the Motion Vector Field (MVF) output by BME. In our work, a cellular automaton is constructed to deduce MV outliers according to a defined local evolution rule. By performing CA‐based evolution in a loop iteration, we gradually expose MV outliers and reduce incorrect MVs resulting from oversmoothing as many as possible. Experimental results show the proposed algorithm can improve the accuracy of BME and provide better objective and subjective interpolation qualities when compared with the traditional MVS algorithms.

Online Digital Image Stabilization for an Unmanned Aerial Vehicle (UAV)

The Unmanned Aerial Vehicle (UAV) video system uses a portable camera mounted on the robot to monitor scene activities. In general, UAVs have very little stabilization equipment, so getting good and stable images of UAVs in real-time is still a challenge. This paper presents a novel framework for digital image stabilization for online applications using a UAV. This idea aims to solve the problem of unwanted vibration and motion when recording video using a UAV. The proposed method is based on dense optical flow to select features representing the displacement of two consecutive frames. K-means clustering is used to find the cluster of the motion vector field that has the largest members. The centroid of the largest cluster was chosen to estimate the rigid transform motion that handles rotation and translation. Then, the trajectory is compensated using the Kalman filter. The experimental results show that the proposed method is suitable for online video stabilization and achieves an average computation time performance of 47.5 frames per second (fps).

Deformable MR-CBCT prostate registration using biomechanically constrained deep learning networks.

Radiotherapeutic dose escalation to dominant intraprostatic lesions (DIL) in prostate cancer could potentially improve tumor control. The purpose of this study was to develop a method to accurately register multiparametric magnetic resonance imaging (MRI) with CBCT images for improved DIL delineation, treatment planning, and dose monitoring in prostate radiotherapy. We proposed a novel registration framework which considers biomechanical constraint when deforming the MR to CBCT. The registration framework consists of two segmentation convolutional neural networks (CNN) for MR and CBCT prostate segmentation, and a three-dimensional (3D) point cloud (PC) matching network. Image intensity-based rigid registration was first performed to initialize the alignment between MR and CBCT prostate. The aligned prostates were then meshed into tetrahedron elements to generate volumetric PC representation of the prostate shapes. The 3D PC matching network was developed to predict a PC motion vector field which can deform the MRI prostate PC to match the CBCT prostate PC. To regularize the network's motion prediction with biomechanical constraints, finite element (FE) modeling-generated motion fields were used to train the network. MRI and CBCT images of 50 patients with intraprostatic fiducial markers were used in this study. Registration results were evaluated using three metrics including dice similarity coefficient (DSC), mean surface distance (MSD), and target registration error (TRE). In addition to spatial registration accuracy, Jacobian determinant and strain tensors were calculated to assess the physical fidelity of the deformation field. The mean and standard deviation of our method were 0.93±0.01, 1.66±0.10mm, and 2.68±1.91mm for DSC, MSD, and TRE, respectively. The mean TRE of the proposed method was reduced by 29.1%, 14.3%, and 11.6% as compared to image intensity-based rigid registration, coherent point drifting (CPD) nonrigid surface registration, and modality-independent neighborhood descriptor (MIND) registration, respectively. We developed a new framework to accurately register the prostate on MRI to CBCT images for external beam radiotherapy. The proposed method could be used to aid DIL delineation on CBCT, treatment planning, dose escalation to DIL, and dose monitoring.

Visualization by P-flow: gradient- and feature-based optical flow and vector fields extracted from image analysis.

We proposed a method for extracting the optical flow suitable for visualization, pseudo-flow (P-flow), from a natural movie [Exp. Brain Res.237, 3321 (2019)EXBRAP0014-481910.1007/s00221-019-05674-0]. The P-flow algorithm comprises two stages: (1) extraction of a local motion vector field from two successive frames and (2) tracking of vectors between two successive frame pairs. In this study, we show that while P-flow takes a feature (vector) tracking approach, it is also classified as a gradient-based approach that satisfies the brightness constancy constraint. We also incorporate interpolation and a corner detector to address the shortcomings associated with the two approaches.

High Frame Rate Video Reconstruction and Deblurring Based on Dynamic and Active Pixel Vision Image Sensor

Active image sensors can only record scenes at low frame rates. The dynamic vision sensor (DVS) can capture high-speed scenes under challenging lighting conditions. However, DVS discards absolute light intensity information. The dynamic and active pixel vision image sensor (DAVIS) can output events and low-frame-rate grayscale images simultaneously. This paper proposes a high frame rate video reconstruction and deblurring algorithm based on events and low-speed blurred image sequences from DAVIS. First, we visualized events as a sequence of event images and performed image denoising process. Then, we used the extended 3D partial recursive search (E-3DPRS) method to generate high-performance motion vector field (MVF) between adjacent denoising event images with low computational complexity. MVF reflects the object motion trajectory with high time resolution. Furthermore, MVF is used as a priori information to restore potentially sharp images, which alleviates the highly ill-posed deblurring process. At last, we reconstructed a high frame rate sharp video based on a sharp image sequence and MVFs. Extensive experiments show that our proposed algorithm outperforms the existing state-of-the-art methods and achieves better average feature similarity and structural similarity.

Spatio-temporal Saliency-based Motion Vector Refinement for Frame Rate Up-conversion

A spatio-temporal saliency-based frame rate up-conversion (FRUC) approach is proposed, which achieves better quality of interpolated frames and invalidates existing texture variation-based FRUC detectors. A spatio-temporal saliency model is designed to select salient frames. After obtaining initial motion vector field by texture- and color-based bilateral motion estimation, two motion vector refining (MVR) schemes are adopted for high and low saliency frames to hierarchically refine the motion vectors, respectively. To produce high-quality interpolated frames, image enhancement are performed for salient frames after frame interpolation. Due to distinct MVR schemes, there are different degrees of texture information in interpolated frames. Some edge and texture information is supplemented into salient frames as post-processing, which can invalidate existing texture variation-based FRUC detectors. Experimental results show that the proposed approach outperforms state-of-the-art works in both objective and subjective qualities of interpolated frames, and achieves the purpose of FRUC anti-forensics.

Very Short-Term Surface Solar Irradiance Forecasting Based On FengYun-4 Geostationary Satellite.

An algorithm to forecast very short-term (30–180 min) surface solar irradiance using visible and near infrared channels (AGRI) onboard the FengYun-4A (FY-4A) geostationary satellite was constructed and evaluated in this study. The forecasting products include global horizontal irradiance (GHI) and direct normal irradiance (DNI). The forecast results were validated using data from Chengde Meteorological Observatory for four typical months (October 2018, and January, April, and July 2019), representing the four seasons. Particle Image Velocimetry (PIV) was employed to calculate the cloud motion vector (CMV) field from the satellite images. The forecast results were compared with the smart persistence (SP) model. A seasonal study showed that July and April forecasting is more difficult than during October and January. For GHI forecasting, the algorithm outperformed the SP model for all forecasting horizons and all seasons, with the best result being produced in October; the skill score was greater than 20%. For DNI, the algorithm outperformed the SP model in July and October, with skill scores of about 12% and 11%, respectively. Annual performances were evaluated; the results show that the normalized root mean square error (nRMSE) value of GHI for 30–180 min horizon ranged from 26.78 to 36.84%, the skill score reached a maximum of 20.44% at the 30-min horizon, and the skill scores were all above 0 for all time horizons. For DNI, the maximum skill score was 6.62% at the 180-min horizon. Overall, compared with the SP model, the proposed algorithm is more accurate and reliable for GHI forecasting and slightly better for DNI forecasting.

Camera operation estimation from video shot using 2D motion vector histogram

This paper presents a novel technique for classifying several camera operations in videos. First of all, we obtain a series of 2D motion vector (MV) fields by applying an existing MV estimation method. Then, a 2D MV histogram is generated in polar coordinates. The histogram shows that how many MVs in each frame share the similar magnitude and orientation. These two MV features are utilized simultaneously to classify the camera operations by representing on the 2D histogram. The proposed method can detect not only single camera operations but also a combination of two camera operations. The 2D histogram can describe the speed of the camera operations. Moreover, the 2D MV field itself can separate zoom-in and zoom-out camera operations that may produce exactly the same pattern in the 2D MV histogram. Especially, separating zoom-in and zoom-out camera operations because these two operations produce a similar 2D histogram. The proposed method can achieve a processing time of 5-10 millisecond per frame for a low-resolution video, while it takes 40-80 millisecond for a high-resolution video.

A Low-Complex Frame Rate Up-Conversion with Edge-Preserved Filtering

The improvement of resolution of digital video requires a continuous increase of computation invested into Frame Rate Up-Conversion (FRUC). In this paper, we combine the advantages of Edge-Preserved Filtering (EPF) and Bidirectional Motion Estimation (BME) in an attempt to reduce the computational complexity. The inaccuracy of BME results from the existing similar structures in the texture regions, which can be avoided by using EPF to remove the texture details of video frames. EPF filters out by the high-frequency components, so each video frame can be subsampled before BME, at the same time, with the least accuracy degradation. EPF also preserves the edges, which prevents the deformation of object in the process of subsampling. Besides, we use predictive search to reduce the redundant search points according to the local smoothness of Motion Vector Field (MVF) to speed up BME. The experimental results show that the proposed FRUC algorithm brings good objective and subjective qualities of the interpolated frames with a low computational complexity.

How to Estimate Global Motion Non-Iteratively From a Coarsely Sampled Motion Vector Field

Nowadays, as the amount of video content increases, the importance of global motion estimation is increasing more and more. This paper considers motion estimation within a compressed domain and presents a robust non-iterative global motion estimation algorithm that efficiently eliminates outliers. The said algorithm is comprised of two parts. The first part uses the distinct-outlier mask based on statistical analysis to eliminate the distinct outliers and uses the current-object mask based on the reference-object mask to eliminate the object outliers. In the second part, the proposed median error designed to be robust to the outlier ratio has been used to estimate weights with higher values as the motion vector gets closer to global motion. Using these weights, the proposed algorithm can accurately estimate global motion non-iteratively, although the outlier ratio may change. The simulation results demonstrate that the proposed algorithm demonstrates the highest estimation accuracy and processing speed in synthetic motion fields as well as real video sequences.