Local Motion Estimation Research Articles

Improved Sea Level Reconstruction from 1900 to 2019

Abstract Before the satellite era, global sea level reconstructions depended on tide gauge records and in situ hydrographic observations. However, the available global mean sea level (GMSL) reconstructions, using different methods, indicate a spread in sea level trend over 1900–2008 (1.3 ∼ 2.0 mm yr−1). With better understanding of the causes of sea level change, here we implement an improved sea level reconstruction, building upon previous work by Church and White, and include three additional factors: the sea level fingerprints, the sterodynamic sea level (SDSL) climate change patterns, and more complete local vertical land motion (VLM) estimates. The trend of new GMSL reconstruction is 1.6 ± 0.2 mm yr−1 (90% confidence level) over 1900–2019, consistent with the sum of observation-based sea level contributions of 1.5 ± 0.2 mm yr−1. The lower trend from the new reconstruction compared with the earlier Church and White result is mainly due to the updated VLM correction. The inclusion of sea level fingerprints and SDSL climate change patterns are the dominant contributors for the improved skill of regional reconstruction. Despite GMSL budget closure in terms of long-term trend since 1900, our study shows discrepancies between the trends from available GMSL reconstructions and the sum of independent observation-based contributions over different periods in the twentieth century, e.g., the discrepancy at the beginning of the twentieth century, which could be related to possible bias in the land ice component estimate. The reconstruction methodology developed here, as tested with synthetic sea level fields, could provide a promising way to identify potential biases in the individual sea level components constrained by available global tide gauge observations.

MoLO: Drift‐free lidar odometry using a 3D model

AbstractLiDAR odometry enables localising vehicles and robots in the environments where global navigation satellite systems (GNSS) are not available. An inherent limitation of LiDAR odometry is the accumulation of local motion estimation errors. Current approaches heavily rely on loop closure to optimise the estimated sensor poses and to eliminate the drift of the estimated trajectory. Consequently, these systems cannot perform real‐time localization and are therefore not practical for a navigation task. This paper presents MoLO, a novel model‐based LiDAR odometry approach to achieve real‐time and drift‐free localization using a 3D model of the environment containing planar surfaces, namely the structural elements of buildings. The proposed approach uses a 3D model of the environment to initialise the LiDAR pose and includes a scan‐to‐scan registration to estimate the pose for consecutive LiDAR scans. Re‐registering LiDAR scans to the 3D model at a certain frequency provides the global sensor pose and eliminates the drift of the trajectory. Pose graphs are built frequently to acquire a smooth and accurate trajectory. A geometry‐based method and a learning‐based method to register LiDAR scans with the 3D model are tested and compared. Experimental results show that MoLO can eliminate drift and achieve real‐time localization while providing an accuracy equivalent to loop closure optimization.

GLOCAL: A self-supervised learning framework for global and local motion estimation

Motions in videos are typically a mixture of local dynamic object motions and global camera motion, which are inconsistent in some cases, and even interfere with each other, causing difficulties in various downstream applications, such as video stabilization that requires the global motion, and action recognition that consumes local motions. Therefore, it is crucial to estimate them separately. Existing methods separate two motions from the mixed motion fields, such as optical flow. However, the quality of mixed motion determines the higher bounds of the performance. In this work, we propose a framework, GLOCAL, to directly estimate global and local motions simultaneously from adjacent frames in a self-supervised manner. Our GLOCAL consists of a Global Motion Estimation (GME) module and a Local Motion Estimation (LME) module. The GME module involves a mixed motion estimation backbone, an implicit bottleneck structure for feature dimension reduction, and an explicit bottleneck for global motion recovery based on the global motion bases with foreground mask under the training guidance of proposed global reconstruction loss. An attention U-Net is adopted for LME which produces local motions while excluding motion of irrelevant regions under the guidance of proposed local reconstruction loss. Our method can achieve better performance than the existing methods on the homography estimation dataset DHE and the action recognition dataset NCAA and UCF-101.

Behaviour Detection and Recognition of College Basketball Players Based on Multimodal Sequence Matching and Deep Neural Networks.

This study fuses multimodal sequence matching with a deep neural network algorithm for college basketball player behavior detection and recognition to conduct in-depth research and analysis, analyzing the basic components of basketball technical action videos by studying the practical application of technical actions in professional games and teaching videos from self-published authors of short video platforms. The characteristics of the dataset are also analyzed through literature research related to the basketball action dataset. On the established basketball technical action dataset, combined with the SSD target detection algorithm, the video images with cropping of human motion regions to reduce the size of image frames, generating a basketball technical action dataset based on cropped frames, reduces the amount of network training and improves the efficiency of subsequent action recognition training. In this study, by analyzing the characteristics of basic camera motion, a univariate global motion model is proposed to introduce a quadratic term to accurately express the shaking transformation, while the horizontal and vertical motion are independently represented to reduce the model complexity. Comparative experimental results show that the proposed model achieves a good balance between complexity and accuracy of global motion representation. It is suitable for global motion modeling in behavior recognition applications, laying the foundation for global and local motion estimation. On this basis, the visual feature change pattern of the key area of the scene (basketball area) is combined with the group behavior recognition based on motion patterns and the success-failure classification based on key visual information to achieve basketball semantic event recognition. The experimental results at NCAA show that the fusion of global and local motion patterns can effectively improve group behavior recognition performance. The semantic event recognition algorithm combining motion patterns and video key visual information achieves the best performance.

USPoint: Self-Supervised Interest Point Detection and Description for Ultrasound-Probe Motion Estimation During Fine-Adjustment Standard Fetal Plane Finding.

Ultrasound (US)-probe motion estimation is a fundamental problem in automated standard plane locating during obstetric US diagnosis. Most recent existing recent works employ deep neural network (DNN) to regress the probe motion. However, these deep regressionbased methods leverage the DNN to overfit on the specific training data, which is naturally lack of generalization ability for the clinical application. In this paper, we are back to generalized US feature learning rather than deep parameter regression. We propose a self-supervised learned local detector and descriptor, named USPoint, for US-probe motion estimation during the fine-adjustment phase of fetal plane acquisition. Specifically, a hybrid neural architecture is designed to simultaneously extract a local feature, and further estimate the probe motion. By embedding a differentiable USPoint-based motion estimation inside the proposed network architecture, the USPoint learns the keypoint detector, scores and descriptors from motion error alone, which doesn't require expensive human-annotation of local features. The two tasks, local feature learning and motion estimation, are jointly learned in a unified framework to enable collaborative learning with the aim of mutual benefit. To the best of our knowledge, it is the first learned local detector and descriptor tailored for the US image. Experimental evaluation on real clinical data demonstrates the resultant performance improvement on feature matching and motion estimation for potential clinical value. A video demo can be found online: https://youtu.be/JGzHuTQVlBs.

Lung tumor segmentation in 4D CT images using motion convolutional neural networks.

Manual delineation on all breathing phases of lung cancer 4D CT image datasets can be challenging, exhaustive, and prone to subjective errors because of both the large number of images in the datasets and variations in the spatial location of tumors secondary to respiratory motion. The purpose of this work is to present a new deep learning-based framework for fast and accurate segmentation of lung tumors on 4D CT image sets. The proposed DL framework leverages motion region convolutional neural network (R-CNN). Through integration of global and local motion estimation network architectures, the network can learn both major and minor changes caused by tumor motion. Our network design first extracts tumor motion information by feeding 4D CT images with consecutive phases into an integrated backbone network architecture, locating volume-of-interest (VOIs) via a regional proposal network and removing irrelevant information via a regional convolutional neural network. Extracted motion information is then advanced into the subsequent global and local motion head network architecture to predict corresponding deformation vector fields (DVFs) and further adjust tumor VOIs. Binary masks of tumors are then segmented within adjusted VOIs via a mask head. A self-attention strategy is incorporated in the mask head network to remove any noisy features that might impact segmentation performance. We performed two sets of experiments. In the first experiment, a five-fold cross-validation on 20 4D CT datasets, each consisting of 10 breathing phases (i.e., 200 3D image volumes in total). The network performance was also evaluated on an additional unseen 200 3D images volumes from 20 hold-out 4D CT datasets. In the second experiment, we trained another model with 40 patients' 4D CT datasets from experiment 1 and evaluated on additional unseen nine patients' 4D CT datasets. The Dice similarity coefficient (DSC), center of mass distance (CMD), 95th percentile Hausdorff distance (HD95 ), mean surface distance (MSD), and volume difference (VD) between the manual and segmented tumor contour were computed to evaluate tumor detection and segmentation accuracy. The performance of our method was quantitatively evaluated against four different methods (VoxelMorph, U-Net, network without global and local networks, and network without attention gate strategy) across all evaluation metrics through a paired t-test. The proposed fully automated DL method yielded good overall agreement with the ground truth for contoured tumor volume and segmentation accuracy. Our model yielded significantly better values of evaluation metrics (p<0.05) than all four competing methods in both experiments. On hold-out datasets of experiment 1 and 2, our method yielded DSC of 0.86 and 0.90 compared to 0.82 and 0.87, 0.75 and 0.83, 081 and 0.89, and 0.81 and 0.89 yielded by VoxelMorph, U-Net, network without global and local networks, and networks without attention gate strategy. Tumor VD between ground truth and our method was the smallest with the value of 0.50 compared to 0.99, 1.01, 0.92, and 0.93 for between ground truth and VoxelMorph, U-Net, network without global and local networks, and networks without attention gate strategy, respectively. Our proposed DL framework of tumor segmentation on lung cancer 4D CT datasets demonstrates a significant promise for fully automated delineation. The promising results of this work provide impetus for its integration into the 4D CT treatment planning workflow to improve the accuracy and efficiency of lung radiotherapy.

Multi-Sensor Fusion for Navigation and Mapping in Autonomous Vehicles: Accurate Localization in Urban Environments

The combination of data from multiple sensors, also known as sensor fusion or data fusion, is a key aspect in the design of autonomous robots. In particular, algorithms able to accommodate sensor fusion techniques enable increased accuracy, and are more resilient against the malfunction of individual sensors. The development of algorithms for autonomous navigation, mapping and localization have seen big advancements over the past two decades. Nonetheless, challenges remain in developing robust solutions for accurate localization in dense urban environments, where the so-called last-mile delivery occurs. In these scenarios, local motion estimation is combined with the matching of real-time data with a detailed pre-built map. In this paper, we utilize data gathered with an autonomous delivery robot to compare different sensor fusion techniques and evaluate which are the algorithms providing the highest accuracy depending on the environment. The techniques we analyze and propose in this paper utilize 3D lidar data, inertial data, GNSS data and wheel encoder readings. We show how lidar scan matching combined with other sensor data can be used to increase the accuracy of the robot localization and, in consequence, its navigation. Moreover, we propose a strategy to reduce the impact on navigation performance when a change in the environment renders map data invalid or part of the available map is corrupted.

Motion estimation of high density crowd using fluid dynamics

ABSTRACT Motion estimation (ME) being a fundamental process of crowd behavior analysis experienced real challenges at high densities due to visual ambiguities and occlusion problems etc. Various surveys reported in the past years summarize conventional ME methods for crowd behaviors at low/medium densities. In this paper, we focus on state-of-the-art fluid dynamics (FD) ME methods developed over the last one and the half-decade for high-density crowd analysis. A detailed discussion is provided on the development of FD ME methods explaining the strengths and weaknesses and viability of FD ME methods for anomaly detection at high crowd densities. Comprehensive experiments are performed comparing the performance of conventional and FD ME at varying crowd densities. Experimentation results show that conventional ME methods fail at high-density crowd whereas FD ME methods could estimate motion only at the global level. Still, research is required to meet the challenges of local ME at high crowd densities.

ADEQUATE IMPROVEMENT FOR SPATIAL VIDEO SCALING FOR VIDEO SURVEILLANCE APPLICATIONS

Spatial Video Scaling (SVS) is a global standard method for video Surveillance Applications. The proposed Spatial Video Scaling (SVS) is designed to estimate the objects in the occupied area and its interconnected vectors. By introducing this SVS framework based interconnected motion vector used to convert the Three-dimensional (3-D) space to the two-dimensional image plane (2-D) at sufficient distance. It consists of several stages of video enhancement and shows an overview of the proposed spatial video scaling (SVS) method. The proposed SVS method is suitable for applications where the accuracy of motion estimation is vital. The proposed SVS video encoding is offered in four phases. The primary phase is the pre-processing stage; in this, unnecessary background areas were removed. Then the Local motion Estimation vectors and global motion Estimation vectors are calculated for better elements detection. Then the extraction of features with Spatial Video Scaling based video enhancement and detection technique. Finally, the performance factors is calculated based on the best features that are found. Finally, the sum of the objects in the movement of the image is smoothed, and the video is compensated.

A gradient-based optical-flow cardiac motion estimation method for cine and tagged MR images.

A new method is proposed to quantify the myocardial motion from both 2D C(ine)-MRI and T(agged)-MRI sequences. The tag pattern offers natural landmarks within the image that makes it possible to accurately quantify the motion within the myocardial wall. Therefore, several methods have been proposed for T-MRI. However, the lack of salient features within the cardiac wall in C-MRI hampers local motion estimation. Our method aims to ensure the local intensity and shape features invariance during motion through the iterative minimization of a cost function via a random walk scheme. The proposed approach is evaluated on realistic simulated C-MRI and T-MRI sequences. The results show more than 53% improvements on displacement estimation, and more than 24% on strain estimation for both C-MRI and T-MRI sequences, as compared to state-of-the-art cardiac motion estimators. Preliminary experiments on clinical data have shown a good ability of the proposed method to detect abnormal motion patterns related to pathology. If those results are confirmed on large databases, this would open up the possibility for more accurate diagnosis of cardiac function from standard C-MRI examinations and also the retrospective study of prior studies.

Weak-structure-aware visual object tracking with bottom-up and top-down context exploration

It poses great challenges to model-free trackers that the object undergoes large appearance variations due to motion, shape deformation, occlusion and surrounding environments. In this paper, we investigate a novel method for modeling and locating the object by being aware of the weak structures of discriminative parts of both the object and its surroundings. The discriminative parts are modeled based on keypoints and feature descriptors. We separate the discriminative parts into two sets corresponding to object and background, and model their spatial structure relationship with the object. While tracking, the successfully localized parts will contribute to potential centers of the object. Aware of the weak structures, we further cluster potential centers to locate the object. The object scale is also updated adaptively. To increase the accuracy of this weak-structure-aware location inference, we fully explore context in both bottom-up and top-down procedures. In the bottom-up stage, we explore the local motion estimation of low-level pixels. The bottom-up information produces consistent tracking of discriminative parts. In the top-down stage, we build a superpixel kernel model to roughly distinguish the object from its surroundings, which provides guided information for location inference and model update. The effectiveness of the proposed method is verified by evaluation on a popular benchmark and comparison with recent tracking methods.

Motion Detection in High Resolution Enhancement

Shifted Superposition (SSPOS) is a resolution enhancement methodwhere apparent high-resolution content is displayed using a lowresolutionprojection system with an opto-mechanical shifter. WhileSSPOS-enhanced projectors have been showing promising resultsin still images, they still suffer from motion artifacts in video contents.Motivated by this, we present a novel approach to apparentprojector resolution enhancement for videos via motion-basedblurring module. We propose the use of a motion detection moduleand a blurring module to compensate for both SSPOS-resulted andnatural motion artifacts in the video content. To accomplish this,we combine both local and global motion estimation algorithms togenerate accurate dense flow fields. The detected motion regionsare enhanced using directional Gaussian filters. Preliminary resultsshow that the proposed method can produce accurate densemotion vectors and significantly reduce the artifacts in videos.

Video pose estimation with global motion cues

This paper addresses the problem of pose estimation in video sequences in which human pose changes drastically over time. Popular strategies for video pose estimation first yield multiple pose candidates for each frame and then achieve consistent pose estimation by enforcing temporal constraints across frames. To enrich pose candidates, previous methods typically employ local motion cues to propagate pose detections to adjacent frames. Reasonable pose proposals can be achieved only when the local motion estimation is accurate and good detections exist among adjacent frames, both of which are hard to be satisfied under drastic human pose changes. In this paper, we propose to propagate pose detections to entire video sequence through global motion cues which provide a long term holistic non-rigid motion transformation for the given video. We exploit the temporal continuity of both single parts and part pairs in the inference over a spatio-temporal model to stitch the reasonable trajectory fragments for each part and obtain the final pose estimation. Experimental results demonstrate remarkable performance improvement in comparison with the state-of-the-art methods.

Human action recognition using histogram of motion intensity and direction from multiple views

This study presents a human action recognition system from multi‐view image sequences. The authors’ approach to human action recognition is based on an estimation of local motion from multiple camera views. The authors propose a new motion descriptor, called histogram of motion intensity and direction, to capture local motion characteristics of human activity. After image normalisation, they estimate motion flow using dense optical flow. Using regular grids, they extract local flow motion and estimate the dominant angle and the intensity of optical flow. The histogram of the dominant angle and its intensity are used as a descriptor for each sequence. After the identification of head direction, they concatenate descriptors in each view as a single feature vector from multiple‐view sequences. Classification based on the proposed feature vector using support vector machine shows better performance than three‐dimensional optical flow‐based approaches, but with lower computational requirements. The authors evaluated action recognition on the publicly available i3DPost and the Institut de Recherche en Informatique et en Automatique (INRIA) Xmas Motion Acquisition Sequences database. Experimental results show promising state‐of‐the‐art results and validate the advanced performance of the authors’ approach.

치료계획용 4D MDCT와 치료 시 획득한 4D CBCT간 영상정합 및 종양 매칭을 이용한 방사선 치료 시 종양 움직임 추적

During image-guided radiation treatment of lung cancer patients, it is necessary to track the tumor motion because it can change during treatment as a consequence of respiratory motion and cardiac motion. In this paper, we propose a method for tracking the motion of the lung tumors based on the three-dimensional image information from planning 4D MDCT and treatment 4D CBCT images. First, to effectively track the tumor motion during treatment, the global motion of the tumor is estimated based on a tumor-specific motion model obtained from planning 4D MDCT images. Second, to increase the accuracy of the tumor motion tracking, the local motion of the tumor is estimated based on the structural information of the tumor from 4D CBCT images. To evaluate the performance of the proposed method, we estimated the tracking results of proposed method using digital phantom. The results show that the tumor localization error of local motion estimation is reduced by 45% as compared with that of global motion estimation.

Multi-Image Sparse Motion-Invariant Photography

In this paper we describe and verify a method, called SMIP, to circumvent the trade-off between motion blur and noise, specifically for scenes with predominantly two distinct linear motions (sparse motion). This is based on employing image stabilization hardware to track objects during exposure while capturing two images in quick succession. The two images are combined into a single sharp image without segmentation or local motion estimation. We provide a theoretical analysis and simulations to show that the Signal-to-Noise Ratio (SNR) increases up to 20 dB over conventional short-exposure photography. We demonstrate that the proposed method significantly improves the SNR compared to existing methods. Furthermore, we evaluate a proof-of-concept using modified off-the-shelf optical image stabilization hardware to verify the effectiveness of our method in practice, showing a good correspondence between the simulation and practical results.

Fusion of Global and Local Motion Estimation Using Foreground Objects for Distributed Video Coding

The side information (SI) in Distributed Video Coding (DVC) is estimated using the available decoded frames and exploited for the decoding and reconstruction of other frames. The quality of the SI has a strong impact on the performance of DVC. Here, we propose a new approach that combines both global and local SI to improve coding performance. Since the background pixels in a frame are assigned to global estimation and the foreground objects to local estimation, one needs to estimate foreground objects in the SI using the backward and forward foreground objects, the background pixels are directly taken from the global SI. Specifically, elastic curves and local motion compensation are used to generate the foreground objects masks in the SI. Experimental results show that, as far as the rate-distortion performance is concerned, the proposed approach can achieve a PSNR improvement of up to 1.39 dB for a group of picture (GOP) size of 2, and up to 4.73 dB for larger GOP sizes, with respect to the reference DISCOVER codec.

Back-propagation beamformer design for motion estimation in echocardiography.

Transverse oscillation (TO) techniques have shown their potential for improving the accuracy of local motion estimation in the transverse direction (i.e., the direction perpendicular to the beam axis). The conventional design of TOs in linear geometry, which is based on the Fraunhofer approximation, relates point spread function (PSF) and apodization function through a Fourier transform. Motivated by the adaptation of TOs in echocardiography, we propose a specific beamforming approach based on back-propagation (BP) to build TOs in sector-shaped geometry. Numerical simulations and experimental data give a comparison between proposed and conventional beamforming for TOs. The accuracy is first quantified by comparing the generated and theoretical PSF using the root mean square error (RMSE) and shows that BP-based beamforming approximates the desired TOs more closely than the conventional approach. Motion estimation is then evaluated. The axial and lateral displacements are within the range [0-0.6] mm and [0°-6.4°], respectively, which correspond to 0.8 times the axial (0.73 mm) and lateral (8°) wavelengths. The result shows that the proposed method yields a clear improvement for lateral displacements, by reducing the error by 28.6% compared with Fourier transform-based beamforming, while maintaining the same error for axial motion estimation. Experimental measurements are discussed to complete this study and confirm that BP-based beamforming leads to better controlled TO images than conventional Fourier-based beamforming.

Averaging, not internal noise, limits the development of coherent motion processing

The development of motion processing is a critical part of visual development, allowing children to interact with moving objects and navigate within a dynamic environment. However, global motion processing, which requires pooling motion information across space, develops late, reaching adult-like levels only by mid-to-late childhood. The reasons underlying this protracted development are not yet fully understood. In this study, we sought to determine whether the development of motion coherence sensitivity is limited by internal noise (i.e., imprecision in estimating the directions of individual elements) and/or global pooling across local estimates. To this end, we presented equivalent noise direction discrimination tasks and motion coherence tasks at both slow (1.5°/s) and fast (6°/s) speeds to children aged 5, 7, 9 and 11 years, and adults. We show that, as children get older, their levels of internal noise reduce, and they are able to average across more local motion estimates. Regression analyses indicated, however, that age-related improvements in coherent motion perception are driven solely by improvements in averaging and not by reductions in internal noise. Our results suggest that the development of coherent motion sensitivity is primarily limited by developmental changes within brain regions involved in integrating motion signals (e.g., MT/V5).

Speckle reduction in optical coherence tomography by matrix completion using bilateral random projection.

Speckle noise is problematic in optical coherence tomography (OCT) and often obscures the structure details. In this paper, we propose a new method to reduce speckle noise from multiply scanned OCT slices. The proposed method registers the OCT scans using a global alignment followed by a local alignment based on global and local motion estimation. Then low rank matrix completion using bilateral random projection is utilized to estimate the noise and recover the clean image. Experimental results show that the proposed method archives average contrast to noise ratio 14.90, better than 13.78 by the state-of-the-art method used in current OCT machines. The technology can be embedded into current OCT machines to enhance the image quality.