Motion Estimation Research Articles

Deep learning-based 6-DoF visual relocalization assisted simultaneous localization and mapping (SLAM)

Purpose Visual simultaneous localization and mapping (SLAM) methods suffer from accumulated errors, especially in challenging environments without loop closure. By constructing lightweight offline maps and using deep learning (DL)-based technology in the two stages, i.e. image retrieval and feature matching, the goal is to reconstruct the six-degree-of-freedom (6-DoF) relationship between SLAM sequences and map sequences. This study aims to propose a comprehensive coarse-to-fine 6-DoF long-term visual relocalization assisted SLAM method specifically designed for challenging environments, aiming to achieve more accurate pose estimation. Design/methodology/approach First, image global feature matching and patch-level global feature matching are conducted to achieve optimal frame-to-frame matching. Second, a DL network is introduced to extract and match features between the most similar frames, enabling point-to-point motion estimation. Finally, a fast pose graph optimization method is proposed to achieve real-time optimization of the pose in the SLAM sequence. Findings The proposed method has been successfully validated on the real-world FinnForest Dataset and UZH-FPV Drone Racing Dataset. The accuracy of the proposed method is evaluated using absolute positional error and absolute rotational error. Experimental results show that in most cases, there are significant improvements in the root mean square error and the standard deviation of the error in pose estimation, and it performs better than loop closure in terms of accuracy. This indicates that the method has strong generalizability and robustness. Originality/value The main contribution of this study is the proposal of a complete DL-based coarse-to-fine 6-DoF long-term visual relocalization method to assist vSLAM, which demonstrates enhanced robustness and generalizability and can eliminate cumulative errors in pose estimation under challenging environments.

Near-Field Clutter Mitigation in Speckle Tracking Echocardiography.

Near-field (NF) clutter filters are critical for unveiling true myocardial structure and dynamics. Randomized singular value decomposition (rSVD) stands out for its proven computational efficiency and robustness. This study investigates the effect of rSVD-based NF clutter filtering on myocardial motion estimation. In silico, material points and their displacements in a homogeneous medium under uniaxial compressions (0.5% - 9% axial strains at 0.5% increments) were simulated in finite-element models. They were exported to the k-Wave toolbox for simulations of pre- and post-deformed ultrasound images with/ without a realistic phase aberrating layer in a high-contrast diverging wave compounding scheme. In vivo, echocardiograms of 20 normal human hearts were acquired using a coded diverging wave compounding imaging method at 3200 frames/second in the transthoracic apical four-chamber view. Morphological component analysis (MCA), which is also a sparse representation method but computationally intensive, was used for comparison with rSVD. Both rSVD- and MCA-based filters were applied to beamformed ultrasound radio-frequency (RF) data before cross-correlation-based speckle tracking. Contrast-to-noise ratios (CNRs) and root-mean-square deviations (RMSDs) were computed from regions of interest to evaluate NF clutter filtering performance of rSVD and MCA. In silico, 2-D displacements estimated from rSVD-based clutter-reduced image data showed strong agreement with ground truth (R2 of 0.95). In vivo, CNR improvements ranged from 1.02 dB to 17.68 dB, consistently enhancing image quality across all subjects. An improvement of ∼4.9 dB in the apical segments was observed in 80% of cases. Mean RMSDs were below 5.0% for all rSVD-based NF clutter-reduced data. While both rSVD and MCA effectively filtered NF clutter, rSVD was significantly more practical. Our findings confirm the reliability, accuracy, and efficiency of rSVD-based clutter filtering in speckle tracking echocardiography. This underscores the feasibility of matrix decomposition-based methods, exemplified by rSVD, in NF clutter filtering for myocardial motion estimation.

Three-dimensional high-isotropic-resolution MR fingerprinting optimized for 0.55 T.

To provide a fast quantitative imaging approach for a 0.55T scanner, where signal-to-noise ratio is limited by the field strength and k-space sampling speed is limited by a lower specification gradient system. We adapted the three-dimensional spiral projection imaging MR fingerprinting approach to 0.55T scanners, with additional features incorporated to improve the image quality of quantitative brain and musculoskeletal imaging, including (i) improved k-space sampling efficiency, (ii) Cramér-Rao lower bound optimized flip-angle pattern for specified T1 and T2 at 0.55 T, (iii) gradient trajectory correction, (iv) attention-based denoising, and (v) motion estimation and correction. The proposed MRF acquisition and reconstruction framework can provide high-quality 1.2-mm isotropic whole-brain quantitative maps and 1-mm isotropic knee quantitative maps, each acquired in 4.5 min. The proposed method was validated in both phantom and in vivo brain and knee studies. By proposing novel methods and integrating advanced techniques, we achieved high-isotropic-resolution MRF on a 0.55T scanner, demonstrating enhanced efficiency, motion resilience, and quantitative accuracy.

Event-Based Visual Simultaneous Localization and Mapping (EVSLAM) Techniques: State of the Art and Future Directions

Recent advances in event-based cameras have led to significant developments in robotics, particularly in visual simultaneous localization and mapping (VSLAM) applications. This technique enables real-time camera motion estimation and simultaneous environment mapping using visual sensors on mobile platforms. Event cameras offer several distinct advantages over frame-based cameras, including a high dynamic range, high temporal resolution, low power consumption, and low latency. These attributes make event cameras highly suitable for addressing performance issues in challenging scenarios such as high-speed motion and environments with high-range illumination. This review paper delves into event-based VSLAM (EVSLAM) algorithms, leveraging the advantages inherent in event streams for localization and mapping endeavors. The exposition commences by explaining the operational principles of event cameras, providing insights into the diverse event representations applied in event data preprocessing. A crucial facet of this survey is the systematic categorization of EVSLAM research into three key parts: event preprocessing, event tracking, and sensor fusion algorithms in EVSLAM. Each category undergoes meticulous examination, offering practical insights and guidance for comprehending each approach. Moreover, we thoroughly assess state-of-the-art (SOTA) methods, emphasizing conducting the evaluation on a specific dataset for enhanced comparability. This evaluation sheds light on current challenges and outlines promising avenues for future research, emphasizing the persisting obstacles and potential advancements in this dynamically evolving domain.

Evaluation of Sea Ice Motion Estimates from Enhanced Resolution Passive Microwave Brightness Temperatures

Sea ice motion plays an important role in the seasonal and interannual evolution of the polar sea ice cover. Satellite imagery can be used to track the motion of sea ice via cross-correlation feature tracking algorithms. Such a method has been used for the National Snow and Ice Data Center (NSIDC) sea ice motion product, based largely on passive microwave imagery. This study investigates the use of a new enhanced resolution passive microwave brightness temperature (TB) product to derive ice motion products. The results demonstrate that the new imagery source provides useful daily motion estimates that provide denser spatial coverage and reduced errors. The enhanced TBs yield motions that have a 30% lower Root Mean Square (RMS) difference with motion estimates from buoys. The enhanced resolution TBs will be used in the new version of the NSIDC motion product that is currently in development.

GNEP based dynamic segmentation and motion estimation for neuromorphic imaging

GNEP based dynamic segmentation and motion estimation for neuromorphic imaging

Uni-DPM: Unifying Self-Supervised Monocular Depth, Pose, and Object Motion Estimation with a Shared Representation

Uni-DPM: Unifying Self-Supervised Monocular Depth, Pose, and Object Motion Estimation with a Shared Representation

GPS/VIO integrated navigation system based on factor graph and fuzzy logic

In today’s technologically advanced landscape, precision in navigation and positioning holds paramount importance across various applications, from robotics to autonomous vehicles. A common predicament in location-based systems is the reliance on Global Positioning System (GPS) signals, which may exhibit diminished accuracy and reliability under certain conditions. Moreover, when integrated with the Inertial Navigation System (INS), the GPS/INS system could not provide a long-term solution for outage problems due to its accumulated errors. This article introduces a novel graph-based method that utilizes a dynamically adjustable fuzzy window to improve navigation and positioning accuracy. This approach effectively integrates GPS data with Visual-Inertial Odometry (VIO). Additionally, it proposes a novel technique for motion estimation and feature extraction, called Adaptive Feature-Flow Fusion, which facilitates robust performance in environments with both high- and low-feature content. The proposed GPS/VIO system is a compelling solution for mitigating GPS-based navigation’s accuracy and reliability concerns. It was implemented and tested on Jetson’s embedded board platform to ensure optimal and real-time system performance. The results from these tests are comprehensively detailed in this article. The system underwent stringent evaluation using the KITTI dataset, demonstrating significant accuracy improvements compared to the Extended Kalman Filter (EKF) system. Specifically, the GPS/VIO system exhibited remarkable accuracy improvements of 84.59% and 88.806% during GPS outages lasting 10 and 27 s, respectively. Furthermore, to enhance precision, data preprocessing techniques were incorporated. These techniques involve optimizing image data by adjusting contrast and brightness levels and applying noise reduction to the Inertial Measurement Unit (IMU) data. This resulted in a substantial 44.7% accuracy enhancement for predefined trajectories.

A study on secure coding of intelligent inspection video in plant areas based on improved deep reinforcement learning

To realize effective data compression, reduce resource occupation, and provide a smoother video transmission experience. The security coding method of an intelligent factory detection video based on improved deep reinforcement learning is studied. First, the transmission and network model of the factory inspection video are analyzed, and a conditional depth network based on Transformer is designed. The network learns the image features through a feature extractor, and processes these features through a motion estimation and compression module. Multi-layer and multi-hypothesis motion compensation and a self-encoder are used to encode features based on the predicted features. The optimal coding strategy is learned using the depth deterministic strategy gradient method, and the residual is outputted in the decoding stage to reconstruct the video frame. The experimental results show that this method can realize the security coding of intelligent inspection video in plant area, and the BDBR (Bjontegaard delta bitrate) index can be saved by 68.01%, and the BD-PSNR (Bjontegaard delta PNSR) index can be increased by 3.49 dB on average; 50 iterations minimize the inefficient strategy ratio; The maximum average return is about 152%; It has small memory occupation, low bandwidth requirements, and high real-time video transmission; High user experience rating.

Data-driven motion estimation for cable-driven end-effectors through parallel 1D-convolution and recurrent neural networks with attention

Minimally invasive surgery requires the inclusion of cable-driven manipulators, due to the advantages in terms of lower inertia, compact design, and remote actuation. However, the nonlinear factors underlying in a cable-driven system bring a challenge for the motion control of the end-effectors of a surgical robot. Further, conventional data-driven approaches for cable-driven systems fail to jointly consider local and temporally sequential information in control signals. To this end, we propose a motion-estimation approach for cable-driven end-effectors, using 1D-convolution and recurrent neural networks with attention, taking into account local and sequential information in modelling control series. Then, we perform cross-domain experiments on our Hefei University of Technology’s Cable-Driven End-effector’s Motion (HFUT-CDEM) dataset, with the experimental results showing that the proposed approach achieves better performance, compared with conventional regression approaches for sequential-data modelling.

Improved deep learning-based IVIM parameter estimation via the use of more "realistic" simulated brain data.

Due to the low signal-to-noise ratio (SNR) and the limited number of b-values, precise parameter estimation of intravoxel incoherent motion (IVIM) imaging remains an open issue to date, especially for brain imaging where the relatively small difference between D and D* easily leads to outliers and obvious graininess in estimated results. To propose a synthetic data driven supervised learning method (SDD-IVIM) for improving precision and noise robustness in IVIM parameter estimation without relying on real-world data for neural network training. On account of the absence of standard IVIM parametric maps from real-world data, a novel model-based method for generating synthetic human brain IVIM data was introduced. Initially, the parameter values of synthetic IVIM parametric maps were sampled from the complex distributions composed of a series of simple and uniform distributions. Subsequently, these parametric maps were modulated with human brain texture to imitate brain tissue structure. Finally, they were used to generate synthetic human brain multi-b-value diffusion-weighted (DW) images based on the IVIM bi-exponential model. With the proposed data synthesis method, an ordinary U-Net with spatial smoothness was employed for IVIM parameter mapping within a supervised learning framework. The performance of SDD-IVIM was evaluated on both numerical phantom and 20 glioma patients. The estimated IVIM parametric maps were compared to those derived from five state-of-the-art methods. In numerical phantom experiments, SDD-IVIM method produces IVIM parametric maps with lower mean absolute error, lower mean bias, and higher structural similarity compared to the other five methods, especially when the SNR of DW images is low. In glioma patient experiments, SDD-IVIM method offers lower coefficient of variation and more reasonable contrast-to-noise ratio between tumor and contralateral normal appearing white matter than the other five methods. Our method owns superior performance in parametric map quality, parameter estimation precision, and lesion characterization in IVIM parameter estimation, with strong resistance to noise.

Improved Sea Level Reconstruction from 1900 to 2019

Abstract Before the satellite era, global sea level reconstructions depended on tide gauge records and in situ hydrographic observations. However, the available global mean sea level (GMSL) reconstructions, using different methods, indicate a spread in sea level trend over 1900–2008 (1.3 ∼ 2.0 mm yr−1). With better understanding of the causes of sea level change, here we implement an improved sea level reconstruction, building upon previous work by Church and White, and include three additional factors: the sea level fingerprints, the sterodynamic sea level (SDSL) climate change patterns, and more complete local vertical land motion (VLM) estimates. The trend of new GMSL reconstruction is 1.6 ± 0.2 mm yr−1 (90% confidence level) over 1900–2019, consistent with the sum of observation-based sea level contributions of 1.5 ± 0.2 mm yr−1. The lower trend from the new reconstruction compared with the earlier Church and White result is mainly due to the updated VLM correction. The inclusion of sea level fingerprints and SDSL climate change patterns are the dominant contributors for the improved skill of regional reconstruction. Despite GMSL budget closure in terms of long-term trend since 1900, our study shows discrepancies between the trends from available GMSL reconstructions and the sum of independent observation-based contributions over different periods in the twentieth century, e.g., the discrepancy at the beginning of the twentieth century, which could be related to possible bias in the land ice component estimate. The reconstruction methodology developed here, as tested with synthetic sea level fields, could provide a promising way to identify potential biases in the individual sea level components constrained by available global tide gauge observations.

Feasibility analysis of utility satellite altimetry and tide gauges for vertical land motion estimation along the coastline of Australia

Feasibility analysis of utility satellite altimetry and tide gauges for vertical land motion estimation along the coastline of Australia

Predicting the high intensity focused ultrasound focus in vivo using acoustic radiation force imaging.

One big challenge in the noninvasive high-intensity focused ultrasound (HIFU) surgery is that the location and shape of its focus is unpredictable at the preoperative stage due to the complexity of sound wave propagation. The Acoustic Radiation Force Impulse (ARFI) imaging is a potential solution to this problem, but artifacts resulting from shear wave propagation remain to be solved. In this study, we proposed avoiding those artefacts by applying the ARFI technique at a high imaging frame rate within a very short time before the shear waves start to propagate. Using single transmission with a convex imaging probe, two ultrafast imaging modalities (the diverging wave and the wide beam), were developed in the ARFI framework, and their reliabilities were validated on a nylon string phantom by the centroid tracking method borrowed from ultrasound localization microscopy (ULM). The proposed ARFI method was tested on a clinically equivalent HIFU system under different acoustic radiation intensities by in-vitro, ex-vivo and in-vivo experiments. In three experimental scenarios, we delivered short HIFU stimulation pulses at varying acoustic powers to induce tissue motion within the focal region. At each experimental site, both diverging wave and wide-beam imaging techniques were employed for motion estimation. Based on the focus prediction derived from the motion estimation, HIFU ablation treatment was performed. The treated samples were then incised to examine the damaged areas. Additionally, ultrasound B-mode images were acquired before and after the procedure and saved for analysis. Quantitative analysis showed that the ARFI with wide beam imaging was able to predict the HIFU focus preoperatively, only with 1 to 3 mm of errors in focal central location, and less than 23% of percentage errors in focal area in most cases. However, the diverging wave imaging failed to predict the HIFU focus due to its low signal-to-noise ratio. In conclusion, the inherent shear wave artefacts in ARFI for predicting the HIFU focus can be successfully avoided by carefully designing the imaging strategy and its working sequence. This ARFI technique was validated through a series of experiments on a clinically equivalent HIFU system, which demonstrated its capability in assisting surgical planning.

Hardware implementation of iterative method for enhanced affine motion estimation in Versatile video coding

Hardware implementation of iterative method for enhanced affine motion estimation in Versatile video coding

Motion-deblurring ghost imaging for an axially moving target.

For lensless ghost imaging (GI) with thermal light, the axially relative motion constrained in the range of the system's depth of focus (DOF) can still cause image blurring because of a variable magnification. We propose a motion-deblurring GI system with pseudo-thermal light, which can overcome the resolution degradation caused by the axial motion. Both the analytical and experimental results demonstrate that high-resolution GI can be always obtained as long as the target's random motion range is smaller than the system's DOF, without using the prior information of motion estimation. We also show that the system's DOF can be extended by optimizing the geometrical shape of the laser spot on the rotating ground glass disk (RGGD). The imaging performance comparison between the proposed GI system and the corresponding lensless GI system is also discussed. This technique can promote the practical application of GI in the field of moving-target detection and recognition.

Self-Attention (SA)-ConvLSTM Encoder–Decoder Structure-Based Video Prediction for Dynamic Motion Estimation

Video prediction, which is the task of predicting future video frames based on past observations, remains a challenging problem because of the complexity and high dimensionality of spatiotemporal dynamics. To address the problems associated with spatiotemporal prediction, which is an important decision-making tool in various fields, several deep learning models have been proposed. Convolutional long short-term memory (ConvLSTM) can capture space and time simultaneously and has shown excellent performance in various applications, such as image and video prediction, object detection, and semantic segmentation. However, ConvLSTM has limitations in capturing long-term temporal dependencies. To solve this problem, this study proposes an encoder–decoder structure using self-attention ConvLSTM (SA-ConvLSTM), which retains the advantages of ConvLSTM and effectively captures the long-range dependencies through the self-attention mechanism. The effectiveness of the encoder–decoder structure using SA-ConvLSTM was validated through experiments on the MovingMNIST, KTH dataset.

Real-time 3D MR guided radiation therapy through orthogonal MR imaging and manifold learning.

In magnetic resonance image (MRI)-guided radiotherapy (MRgRT), 2D rapid imaging is commonly used to track moving targets with high temporal frequency to minimize gating latency. However, anatomical motion is not constrained to 2D, and a portion of the target may be missed during treatment if 3D motion is not evaluated. While some MRgRT systems attempt to capture 3D motion by sequentially tracking motion in 2D orthogonal imaging planes, this approach assesses 3D motion via independent 2D measurements at alternating instances, lacking a simultaneous 3D motion assessment in both imaging planes. We hypothesized that a motion model could be derived from prior 2D orthogonal imaging to estimate 3D motion in both planes simultaneously. We present a manifold learning technique to estimate 3D motion from 2D orthogonal imaging. Five healthy volunteers were scanned under an IRB-approved protocol using a 3.0 T Siemens Skyra simulator. Images of the liver dome were acquired during free breathing (FB) with a 2.6mm × 2.6mm in-plane resolution for approximately 10min in alternating sagittal and coronal planes at ∼5 frames per second. The motion model was derived using a combined manifold learning and alignment approach based on locally linear embedding (LLE). The model utilized the spatially overlapping MRI signal shared by both imaging planes to group together images that had similar signals, enabling motion estimation in both planes simultaneously. The model's motion estimates were compared to the ground truth motion derived in each newly acquired image using deformable registration. A simulated target was defined on the dome of the liver and used to evaluate model performance. The Dice similarity coefficient and distance between the model-tracked and image-tracked contour centroids were evaluated. Motion modeling error was estimated in the orthogonal plane by back-propagating the motion to the currently imaged plane and by interpolating the motion between image acquisitions where ground truth motion was available. The motion observed in the healthy volunteer studies ranged from 12.6 to 38.7mm. On average, the model demonstrated sub-millimeter precision and>0.95 Dice coefficient compared to the ground truth motion observed in the currently imaged plane. The average Dice coefficient and centroid distance between the model-tracked and ground truth target contours were 0.96±0.03 and 0.26mm±0.27mm respectively across all volunteer studies. The out-of-plane centroid motion error was estimated to be 0.85mm±1.07mm and 1.26mm±1.38mm using the back-propagation (BP) and interpolation error estimation methods. The healthy volunteer studies indicate promising results using the proposed motion modeling technique. Out-of-plane modeling error was estimated to be higher but still demonstrated sub-voxel motion accuracy.

Single-pixel imaging robust to arbitrary translational motion.

Single-pixel imaging (SPI) stands out in computational imaging for its simplicity and adaptability, yet its performance has been hampered by artifacts from translational motion. Existing solutions heavily rely on accurate motion modeling, requiring additional hardware and computational costs. In this Letter, we propose translational motion-agnostic SPI (TMA-SPI), a novel, to the best of our knowledge, single-object SPI framework agnostic to arbitrary translational motion. Our dual-domain optimization method leverages the translation invariance property of the amplitude spectrum in the Fourier domain, combined with the spatially finite and nonnegative constraints in the image domain, to produce a clear image of the moving object without any motion estimation or compensation. Through both simulation and the deployment of a real imaging prototype, we demonstrate its superior performance over the conventional SPI method. Our framework is expected to extend the applicability of SPI, offering significant improvements for dynamic sensing applications.

A ViT‐Based Adaptive Recurrent Mobilenet With Attention Network for Video Compression and Bit‐Rate Reduction Using Improved Heuristic Approach Under Versatile Video Coding

ABSTRACTVideo compression received attention from the communities of video processing and deep learning. Modern learning‐aided mechanisms use a hybrid coding approach to reduce redundancy in pixel space across time and space, improving motion compensation accuracy. The experiments in video compression have important improvements in past years. The Versatile Video Coding (VVC) is the primary enhancing standard of video compression which is also referred to as H. 226. The VVC codec is a block‐assisted hybrid codec, making it highly capable and complex. Video coding effectively compresses data while reducing compression artifacts, enhancing the quality and functionality of AI video technologies. However, the traditional models suffer from the incorrect compression of the motion and ineffective compensation frameworks of the motion leading to compression faults with a minimal trade‐off of the rate distortion. This work implements an automated and effective video compression task under VVC using a deep learning approach. Motion estimation is conducted using the Motion Vector (MV) encoder‐decoder model to track movements in the video. Based on these MV, the reconstruction of the frame is carried out to compensate for the motions. The residual images are obtained by using Vision Transformer‐based Adaptive Recurrent MobileNet with Attention Network (ViT‐ARMAN). The parameters optimization of the ViT‐ARMAN is done using the Opposition‐based Golden Tortoise Beetle Optimizer (OGTBO). Entropy coding is used in the training phase of the developed work to find the bit rate of residual images. Extensive experiments were conducted to demonstrate the effectiveness of the developed deep learning‐based method for video compression and bit rate reduction under VVC.