Shaky Motion Research Articles

DTP: learning to estimate full-body pose in real-time from sparse VR sensor measurements

For virtual reality (VR) applications, estimating full-body pose in real-time is becoming increasingly popular. Previous works have reconstructed full-body motion in real time from an HTC VIVE headset and five VIVE Tracker measurements by solving the inverse kinematics (IK) problem. However, an IK solver may yield unnatural poses and shaky motion. This paper introduces Deep Tracker poser (DTP): a method for real-time full-body pose estimation in VR. This task is difficult due to the ambiguous mapping from the sparse measurements to full-body pose. The data obtained from VR sensors is calibrated, normalized and fed into the deep neural networks (DNN). To learn from sufficient data, we propose synthesizing a VR sensor dataset called AMASS-VR from the AMASS, a collection of various motion capture datasets. Furthermore, feet tracking loss is a common problem of VIVE Tracker. To improve the accuracy and robustness of DTP to the occlusion noise, we simulate the occlusion noise by Gaussian random noise. Then we synthesize an occlusion dataset AMASS-OCC and fine-tune DTP on that. We evaluate DTP by comparing with other popular methods in terms of the accuracy and computational cost. The results indicate that DTP outperforms others in terms of the positional error (1.04 cm) and rotational error (4.22 °). The quantitative and qualitative results show that DTP reconstructs accurate and natural full-body pose even under serious feet occlusion, which indicates the superiority of the DTP in modelling the mapping from sparse joint data to the full-body pose.

A Novel Video Stabilization Model With Motion Morphological Component Priors

Video stabilization is the process of improving the video quality by removing annoying fluctuant motion caused by camera jittering. A key issue of a successful solution is the temporal adaptability to motion and the overall robustness with respect to different motion types. However, most previous methods usually produce non-motion adaptive stabilized videos. In other words, under-smoothing in slow motion segments and over-smoothing in rapid motion segments will be produced for complex shaky videos. To overcome these drawbacks, we propose a novel video stabilization approach using a motion morphological component (MMC) decomposition. Specifically, the observed motion is decomposed into three MMCs: low-frequency smoothed (LFS) motion, high-frequency compensatory (HFC) motion, and shaky motion. LFS motion helps to largely stabilize videos, and HFC motion helps to recover missing motion to deal with over-smoothing. Subsequently, we present an MMC-based model to retrieve the desired smoothed motion, in which weighted nuclear norm and autoregression priors are used for LFS motion, while a sparsity prior is adopted for HFC motion. In addition, we design an adaptive weight setting scheme to detect rapid motions and to calculate the optimal weights. Finally, we develop a stabilization algorithm under the Alternating Direction Method of Multipliers (ADMM) framework. Experimental results demonstrate that our method can achieve high-quality results compared with that of other state-of-the-art stabilization methods in terms of robustness and efficiency, both quantitatively and qualitatively.

Content-and-disparity-aware stereoscopic video stabilization

Filming stereoscopic videos has become easier with the development of science and technology, and such videos now proliferate on the Internet. Meanwhile, video stabilization is an important research topic. Thus, this study presents a method of stabilizing stereoscopic videos with preserving the disparities between objects in the frames. First, the feature points must be tracked and separated into many groups. We posit that the shaky motion is caused not only by translations but also by rotations. Thus, directly smoothing the path will not produce a similar trajectory so that we solve the shakiness of the turning before smoothing the path. To address such shakiness, we initially estimate the rotation angles between two adjacent frames. By determining the angle changes of all the frames, we can find out the preference of rotation in a video. Furthermore, the inconsistent angular velocity can be alleviated and the shakiness of the turning is solved by rotating the frame appropriately. Then, the Bezier curve is utilized to smooth the trajectories. We split a trajectory into a set of subtrajectories and subsequently smooth the latter independently. Unlike previous researches, we split the trajectory according to the feature tracking rate to obtain similar trajectories in the original video path. After making subtrajectories smooth, we merge them to attain a smoothed trajectory. The joint of the two subtrajectories is replaced by their interpolation. Finally, we optimize the smoothness and context preservation to stabilize videos without requiring extensive clipping.

EyeSAM: graph-based localization and mapping of retinal vasculature during intraocular microsurgery.

Robot-assisted intraocular microsurgery can improve performance by aiding the surgeon in operating on delicate micron-scale anatomical structures of the eye. In order to account for the eyeball motion that is typical in intraocular surgery, there is a need for fast and accurate algorithms that map the retinal vasculature and localize the retina with respect to the microscope. This work extends our previous work by a graph-based SLAM formulation using a sparse incremental smoothing and mapping (iSAM) algorithm. The resulting technique, "EyeSAM," performs SLAM for intraoperative vitreoretinal surgical use while avoiding spurious duplication of structures as with the previous simpler technique. The technique also yields reduction in average pixel error in the camera motion estimation. This work provides techniques to improve intraoperative tracking of retinal vasculature by handling loop closures and achieving increased robustness to quick shaky motions and drift due to uncertainties in the motion estimation.

Geometry of Motion for Video Shakiness Detection

This paper presents a novel algorithm for automatically detecting global shakiness in casual videos. Perframe amplitude is computed by the geometry of motion, based on the kinematic model defined by inter-frame geometric transformations. Inspired by motion perception, we investigate the just-noticeable amplitude of shaky motion perceived by the human visual system. Then, we use the thresholding contrast strategy on the statistics of per-frame amplitudes to determine the occurrence of perceived shakiness. For testing the detection accuracy, a dataset of video clips is constructed with manual shakiness label as the ground truth. The experiments demonstrate that our algorithm can obtain good detection accuracy that is in concordance with subjective judgement on the videos in the dataset.

EyeSLAM: Real-time simultaneous localization and mapping of retinal vessels during intraocular microsurgery.

Fast and accurate mapping and localization of the retinal vasculature is critical to increasing the effectiveness and clinical utility of robot-assisted intraocular microsurgery such as laser photocoagulation and retinal vessel cannulation. The proposed EyeSLAM algorithm delivers 30Hz real-time simultaneous localization and mapping of the human retina and vasculature during intraocular surgery, combining fast vessel detection with 2D scan-matching techniques to build and localize a probabilistic map of the vasculature. In the harsh imaging environment of retinal surgery with high magnification, quick shaky motions, textureless retina background, variable lighting and tool occlusion, EyeSLAM can map 75% of the vessels within two seconds of initialization and localize the retina in real time with a root mean squared (RMS) error of under 5.0 pixels (translation) and 1° (rotation). EyeSLAM robustly provides retinal maps and registration that enable intelligent surgical micromanipulators to aid surgeons in simulated retinal vessel tracing and photocoagulation tasks.

A Global Approach to Fast Video Stabilization

This paper presents a novel formulation of video stabilization by directly solving for optimal image warps toward stabilized sequence. With the estimated shaky motion via long or short feature trajectories, our approach encodes another two steps, motion compensation and image warping, into a single global optimization process, rather than operating as two individual steps. This process is done only with positions of embedded mesh vertices as common variables. Spatial and temporal coherence is therein reformulated with similarity-invariant representation of motion trajectories and intra- (and inter-) frame consistency of similar transformations with respect to mesh vertices. Such a one-shot formulation converts video stabilization into a quadratic energy minimization problem defined for image warps, and thus can be efficiently resolved by using a robust solver for sparse linear systems. Experimental results demonstrate the flexibility and efficiency of our approach in producing visually plausible stabilization effects on a variety of videos.

Fast Image Stitching For Video Stabilization Using Sift Feature Points

Video Stabilization For Vehicular Applications Is An Important Method Of Removing Unwanted Shaky Motions From Unstable Videos. In This Paper, An Improved Video Stabilization Method With Image Stitching Has Been Proposed. Scale Invariant Feature Transform (Sift) Matching Is Used To Calculate The New Position Of The Points In Next Frame. Image Stitching Is Done In Every Frame To Get Stabilized Frames To Provide Stable Video As Well As A Better Understanding Of The Previous Frame’S Position And Show The Surrounding Objects Together. The Computational Complexity Of Sift (Scale-Invariant Feature Transform) Is Reduced By Reducing The Sift Descriptors Size And Resticting The Number Of Keypints To Be Extracted. Also, A Modified Matching Procedure Is Proposed To Improve The Accuracy Of The Stabilization.

Object-Based Rendering and 3-D Reconstruction Using a Moveable Image-Based System

This paper proposes a movable image-based rendering (M-IBR) system for improving the viewing freedom and environmental modeling capability of conventional static IBR systems. The system supports object-based rendering and 3-D reconstruction capability and consists of three main components. 1) An improved video stabilization method to reduce the shaky motion frequently encountered in movable IBR systems. It employs local polynomial regression (LPR) to automatically select an appropriate bandwidth for smoothing the estimated motion. 2) A novel view synthesis algorithm using a new segmentation and mutual-information (MI)-based algorithm for dense depth map estimation, which relies on segmentation, LPRbased depth map smoothing, and MI-based matching algorithm to iteratively estimate the depth map. The method is very flexible and both semiautomatic and automatic segmentation methods can be employed. They rank fourth and sixth, respectively, in the Middlebury comparison of existing depth estimation methods. This allows high-quality renderings of outdoor scenes with improved mobility/freedom to be obtained. 3) A new 3-D reconstruction algorithm that utilizes the sequential structure-from-motion technique and the dense depth maps estimated previously. It relies on a new iterative point cloud refinement algorithm based on Kalman filter for outlier removal and the segmentation-MI-based algorithm to further refine the correspondences and the projection matrices. The mobility of our system allows us to recover more conveniently 3-D model of static objects from the improved point cloud using a new robust radial basis function-based modeling algorithm to further suppress possible outliers and generate smooth 3-D meshes of objects. Experimental results show that the proposed 3-D reconstruction algorithm significantly reduces the adverse effect of the outliers and produces high-quality renderings using shadow light field and the model reconstructed.

Capturing Intention‐based Full‐Frame Video Stabilization

AbstractAnnoying shaky motion is one of the significant problems in home videos, since hand shake is an unavoidable effect when capturing by using a hand‐held camcorder. Video stabilization is an important technique to solve this problem, but the stabilized videos resulting from some current methods usually have decreased resolution and are still not so stable. In this paper, we propose a robust and practical method of full‐frame video stabilization while considering user's capturing intention to remove not only the high frequency shaky motions but also the low frequency unexpected movements. To guess the user's capturing intention, we first consider the regions of interest in the video to estimate which regions or objects the user wants to capture, and then use a polyline to estimate a new stable camcorder motion path while avoiding the user's interested regions or objects being cut out. Then, we fill the dynamic and static missing areas caused by frame alignment from other frames to keep the same resolution and quality as the original video. Furthermore, we smooth the discontinuous regions by using a three‐dimensional Poisson‐based method. After the above automatic operations, a full‐frame stabilized video can be achieved and the important regions and objects can also be preserved.

Adaptive Sample Position Control System for Electrostatic Levitation Furnace

The Electrostatic Levitation Furnace (ELF) is one of the experiment facilities for materials science which is expected to be used on the International Space Station (ISS) in future. A unique feature of ELF is to levitate a sample inside the furnace by means of Coulomb's force throughout the fusion/solidification experiments. To realize this condition under the disturbances of the ISS environment we designed three customized control logics. Incidentally, when those logics were tested in a rocket experiment in 1998, the sample material showed an unexpectedly shaky motion during the experiment. The cause of this phenomenon was explained by analysis that the control force diminished when the sample temperature was high and the heating lasers were lit. Actually, the electrical charge of the sample significantly decreases when it is heated to a certain degree and is irradiated by the lasers. Hence, we redesigned the control software by adopting the Recursive Least-Squares Algorithm (RLSA), which identifies the amount of a sample's electrical charge in real-time. As a result, several numerical simulations have proved that the sample motion is well controlled with this adaptive control software even if the electrical charge changes significantly. Consequently, we concluded that this adaptive control technique is widely applicable to the systems whose parameters are changeable during operations.

Full-frame video stabilization with motion inpainting

Video stabilization is an important video enhancement technology which aims at removing annoying shaky motion from videos. We propose a practical and robust approach of video stabilization that produces full-frame stabilized videos with good visual quality. While most previous methods end up with producing smaller size stabilized videos, our completion method can produce full-frame videos by naturally filling in missing image parts by locally aligning image data of neighboring frames. To achieve this, motion inpainting is proposed to enforce spatial and temporal consistency of the completion in both static and dynamic image areas. In addition, image quality in the stabilized video is enhanced with a new practical deblurring algorithm. Instead of estimating point spread functions, our method transfers and interpolates sharper image pixels of neighboring frames to increase the sharpness of the frame. The proposed video completion and deblurring methods enabled us to develop a complete video stabilizer which can naturally keep the original image quality in the stabilized videos. The effectiveness of our method is confirmed by extensive experiments over a wide variety of videos.