Non-rigid Motion Research Articles

Aging and the visual perception of rigid and nonrigid motion

Nonrigid forms of motion are commonplace in everyday life. Given previously documented age-related deteriorations in various tasks involving motion (discriminating speed, identifying motion direction, etc.), an experiment was conducted to evaluate the potential effect of age upon the visual ability to detect rigid and nonrigid object motion. Thirty younger and older observers participated in the experiment (mean ages were 19.9 and 75.8 years, respectively). As has been done multiple times in the past, the individual motions of object vertices were manipulated to simulate either rigid motion (rotation in depth, with or without precession) or two different types of nonrigid motion (also rotation in depth with or without precession, but with added object deformation). In confirmation of previous research, there were large effects of nonrigid motion type and precession upon the ability to differentiate between rigid and nonrigid object motion. There was also a large effect of age, such that the discrimination performance of the younger observers was 49.6% higher than that exhibited by the older observers. In this first ever study of aging and nonrigid object motion perception, we thus find that aging is associated with a substantial impairment in the ability to visually perceive object nonrigidity.

Inability to pursue nonrigid motion produces instability of spatial perception.

Vision generates a stable representation of space by combining retinal input with internal predictions about the visual consequences of eye movements. We report a type of nonrigid motion that disrupts the connection between eye movements and perception, causing visual instability. This motion is accurately perceived during fixation, but it cannot be pursued. Catch-up saccades are accurately directed to the moving target but the motion stimulus appears to jump in space with each saccade. Our results reveal four major findings about perception and the visuomotor system: (i) Pursuit fails for certain types of motion; (ii) pursuit and catch-up saccades are independently controlled; (iii) prediction of saccade consequences is independent from saccade control; and (iv) the visual stability of moving objects relies on similar motion mechanisms as pursuit.

Performance Study of an MRI Motion-Compensated Reconstruction Program on Intel CPUs, AMD EPYC CPUs, and NVIDIA GPUs

Motion-compensated image reconstruction enables new clinical applications of Magnetic Resonance Imaging (MRI), but it relies on computationally intensive algorithms. This study focuses on the Generalized Reconstruction by Inversion of Coupled Systems (GRICS) program, applied to the reconstruction of 3D images in cases of non-rigid or rigid motion. It uses hybrid parallelization with the MPI (Message Passing Interface) and OpenMP (Open Multi-Processing). For clinical integration, the GRICS needs to efficiently harness the computational resources of compute nodes. We aim to improve the GRICS’s performance without any code modification. This work presents a performance study of GRICS on two CPU architectures: Intel Xeon Gold and AMD EPYC. The roofline model is used to study the software–hardware interaction and quantify the code’s performance. For CPU–GPU comparison purposes, we propose a preliminary MATLAB–GPU implementation of the GRICS’s reconstruction kernel. We establish the roofline model of the kernel on two NVIDIA GPU architectures: Quadro RTX 5000 and A100. After the performance study, we propose some optimization patterns for the code’s execution on CPUs, first considering only the OpenMP implementation using thread binding and affinity and appropriate architecture-compilation flags and then looking for the optimal combination of MPI processes and OpenMP threads in the case of the hybrid MPI–OpenMP implementation. The results show that the GRICS performed well on the AMD EPYC CPUs, with an architectural efficiency of 52%. The kernel’s execution was fast on the NVIDIA A100 GPU, but the roofline model reported low architectural efficiency and utilization.

3D B1+ corrected simultaneous myocardial T1 and T1ρ mapping with subject-specific respiratory motion correction and water-fat separation.

To develop a 3D free-breathing cardiac multi-parametric mapping framework that is robust to confounders of respiratory motion, fat, and B1+ inhomogeneities and validate it for joint myocardial T1 and T1ρ mapping at 3T. An electrocardiogram-triggered sequence with dual-echo Dixon readout was developed, where nine cardiac cycles were repeatedly acquired with inversion recovery and T1ρ preparation pulses for T1 and T1ρ sensitization. A subject-specific respiratory motion model relating the 1D diaphragmatic navigator to the respiration-induced 3D translational motion of the heart was constructed followed by respiratory motion binning and intra-bin 3D translational and inter-bin non-rigid motion correction. Spin history B1+ inhomogeneities were corrected with optimized dual flip angle strategy. After water-fat separation, the water images were matched to the simulated dictionary for T1 and T1ρ quantification. Phantoms and 10 heathy subjects were imaged to validate the proposed technique. The proposed technique achieved strong correlation (T1: R2 = 0.99; T1ρ: R2 = 0.98) with the reference measurements in phantoms. 3D cardiac T1 and T1ρ maps with spatial resolution of 2 × 2 × 4 mm were obtained with scan time of 5.4 ± 0.5 min, demonstrating comparable T1 (1236 ± 59 ms) and T1ρ (50.2 ± 2.4 ms) measurements to 2D separate breath-hold mapping techniques. The estimated B1+ maps showed spatial variations across the left ventricle with the septal and inferior regions being 10%-25% lower than the anterior and septal regions. The proposed technique achieved efficient 3D joint myocardial T1 and T1ρ mapping at 3T with respiratory motion correction, spin history B1+ correction and water-fat separation.

Ultrasound localization microscopy based microvasculature evaluation of spinal cord injury with varying severities

Abstract The vasculature plays an important role in spinal cord function. Traumatic spinal cord injury (SCI) often results in a vascular disruption after primary insult, following secondary injury in neighbouring tissue. However, standard angiography methods such as digital subtraction angiography (DSA) still have difficulties in microvasculature imaging, especially for spinal cord with small size and non-rigid motion. The ultrasound localization microscopy (ULM) disrupts the compromise between penetration and resolution and provides super-resolution images of microvasculature. In this study, the applicability of the ULM for in vivo SCI evaluation was investigated on rat models with varying severities (different compression time durations correspond to mild and severe injury). Ultrasonic parameters including vessel saturation and blood velocity were measured via ULM. Basso, Beattie, Bresnahan (BBB) locomotion test was applied for injury severity validation and comparison with ultrasonic measurements.

Deep Learning Methods to Analyze the Forces and Torques in Joints Motion

This paper proposes a composite model that combines convolutional neural network models and mechanical analysis to determine the forces acting on an object. First, we establish a model using Newtonian mechanics to analyze the forces experienced by the human body during movement, particularly the forces on joints. The model calculates the mapping relationship between the object’s movement and the forces on the joints. Then, by analyzing a large number of fencing competition videos using a deep learning model, we extract video features to study the torques and forces on human joints. Our analysis of numerous images reveals that, in certain movement patterns, the peak pressure on the knee joint can be two to three times higher than in a normal state, while the driving knee can withstand peak torques of 400–600 Nm. This straightforward model can effectively capture the forces and torques on the human body during movement using a deep neural network. Furthermore, this model can also be applied to problems involving non-rigid body motion.

Repulsive Shells

This paper develops a shape space framework for collision-aware geometric modeling, where basic geometric operations automatically avoid inter-penetration. Shape spaces are a powerful tool for surface modeling, shape analysis, nonrigid motion planning, and animation, but past formulations permit nonphysical intersections. Our framework augments an existing shape space using a repulsive energy such that collision avoidance becomes a first-class property, encoded in the Riemannian metric itself. In turn, tasks like intersection-free shape interpolation or motion extrapolation amount to simply computing geodesic paths via standard numerical algorithms. To make optimization practical, we develop an adaptive collision penalty that prevents mesh self-intersection, and converges to a meaningful limit energy under refinement. The final algorithms apply to any category of shape, and do not require a dataset of examples, training, rigging, nor any other prior information. For instance, to interpolate between two shapes we need only a single pair of meshes with the same connectivity. We evaluate our method on a variety of challenging examples from modeling and animation.

Multi-day neuron tracking in high-density electrophysiology recordings using earth mover’s distance

Accurate tracking of the same neurons across multiple days is crucial for studying changes in neuronal activity during learning and adaptation. Advances in high-density extracellular electrophysiology recording probes, such as Neuropixels, provide a promising avenue to accomplish this goal. Identifying the same neurons in multiple recordings is, however, complicated by non-rigid movement of the tissue relative to the recording sites (drift) and loss of signal from some neurons. Here, we propose a neuron tracking method that can identify the same cells independent of firing statistics, that are used by most existing methods. Our method is based on between-day non-rigid alignment of spike-sorted clusters. We verified the same cell identity in mice using measured visual receptive fields. This method succeeds on datasets separated from 1 to 47 days, with an 84% average recovery rate.

Multi-day neuron tracking in high-density electrophysiology recordings using earth mover's distance.

Accurate tracking of the same neurons across multiple days is crucial for studying changes in neuronal activity during learning and adaptation. Advances in high-density extracellular electrophysiology recording probes, such as Neuropixels, provide a promising avenue to accomplish this goal. Identifying the same neurons in multiple recordings is, however, complicated by non-rigid movement of the tissue relative to the recording sites (drift) and loss of signal from some neurons. Here, we propose a neuron tracking method that can identify the same cells independent of firing statistics, that are used by most existing methods. Our method is based on between-day non-rigid alignment of spike-sorted clusters. We verified the same cell identity in mice using measured visual receptive fields. This method succeeds on datasets separated from 1 to 47 days, with an 84% average recovery rate.

TSUDepth: Exploring temporal symmetry-based uncertainty for unsupervised monocular depth estimation

When faced with occlusions and non-rigid motions, machines often struggle with depth estimation, a task effortlessly performed by humans with just one eye. Continuous RGB images embody rich temporal features, such as symmetry and optical flow, which current deep-learning models fail to effectively leverage. In response to this limitation, we introduce an innovative framework known as Temporal Symmetry-based Uncertainty (TSU)-Depth, aimed at enhancing the accuracy of unsupervised monocular depth estimation. The Temporal Symmetry-based Occlusion Optimization (TSOO) component plays a pivotal role in robustly identifying occluded regions and comparable optimization across adjacent frames. Simultaneously, we propose Temporal Optical Flow Masking (TOFM) to effectively identify and exclude static pixels (such as out-of-range depths and non-rigid objects) between adjacent frames. Additionally, we introduce Cross-Resolution Distillation (CRED) to enhance depth estimation accuracy across various resolutions, especially in low input resolution scenarios. Furthermore, we designed a new depth estimation structure utilizing the DPT structure and incorporating a GRU module to enhance performance details. Through extensive experiments on benchmark datasets, including KITTI, Cityscapes, and Make3D, our TSUDepth framework has consistently demonstrated state-of-the-art performance. Code is available at https://github.com/BlueEg/TSUDepth/.

Perceptual Biases in the Interpretation of Non-Rigid Shape Transformations from Motion.

Most existing research on the perception of 3D shape from motion has focused on rigidly moving objects. However, many natural objects deform non-rigidly, leading to image motion with no rigid interpretation. We investigated potential biases underlying the perception of non-rigid shape interpretations from motion. We presented observers with stimuli that were consistent with two qualitatively different interpretations. Observers were shown a two-part 3D object with the smaller part changing in length dynamically as the whole object rotated back and forth. In two experiments, we studied the misperception (i.e., perceptual reinterpretation) of the non-rigid length change to a part. In Experiment 1, observers misperceived this length change as a part orientation change (i.e., the smaller part was seen as articulating with respect to the larger part). In Experiment 2, the stimuli were similar, except the silhouette of the part was visible in the image. Here, the non-rigid length change was reinterpreted as a rigidly attached part with an "illusory" non-orthogonal horizontal angle relative to the larger part. We developed a model that incorporated this perceptual reinterpretation and could predict observer data. We propose that the visual system may be biased towards part-wise rigid interpretations of non-rigid motion, likely due to the ecological significance of movements of humans and other animals, which are generally constrained to move approximately part-wise rigidly. That is, not all non-rigid deformations are created equal: the visual systems' prior expectations may bias the system to interpret motion in terms of biologically plausible shape transformations.

Analysis of Instantaneous Kinematic Properties Regarding the Shape of Robotic Mechanisms

Abstract A theoretical framework for analyzing instantaneous kinematic properties regarding the shape of robotic mechanisms is proposed. Conventional research on kinematic analysis regarding the shape has been conducted with the primary goal of approximating a target shape. In contrast, this study does not set a target shape, and instantaneous kinematic properties regarding the shape formed by multiple reference links are analyzed. The properties provide information about the shape change that is easy to achieve and about the controllability of the shape. Furthermore, in contrast to the standard kinematic analysis of robot manipulators that assumes having a single hand link configuration with respect to the base link, the kinematic analysis in this study is free from this assumption. When analyzing instantaneous kinematic properties regarding the shape, it is crucial to decompose motions of reference links into rigid and nonrigid components. After formulating this decomposition, the Jacobian matrices that relate active joint velocities and nonrigid motions are defined. The indices of instantaneous kinematic properties regarding the shape are defined on the basis of the Jacobian matrices. Moreover, application examples using the defined indices are demonstrated.

3D cine-magnetic resonance imaging using spatial and temporal implicit neural representation learning (STINR-MR).

Objective. 3D cine-magnetic resonance imaging (cine-MRI) can capture images of the human body volume with high spatial and temporal resolutions to study anatomical dynamics. However, the reconstruction of 3D cine-MRI is challenged by highly under-sampled k-space data in each dynamic (cine) frame, due to the slow speed of MR signal acquisition. We proposed a machine learning-based framework, spatial and temporal implicit neural representation learning (STINR-MR), for accurate 3D cine-MRI reconstruction from highly under-sampled data.Approach. STINR-MR used a joint reconstruction and deformable registration approach to achieve a high acceleration factor for cine volumetric imaging. It addressed the ill-posed spatiotemporal reconstruction problem by solving a reference-frame 3D MR image and a corresponding motion model that deforms the reference frame to each cine frame. The reference-frame 3D MR image was reconstructed as a spatial implicit neural representation (INR) network, which learns the mapping from input 3D spatial coordinates to corresponding MR values. The dynamic motion model was constructed via a temporal INR, as well as basis deformation vector fields (DVFs) extracted from prior/onboard 4D-MRIs using principal component analysis. The learned temporal INR encodes input time points and outputs corresponding weighting factors to combine the basis DVFs into time-resolved motion fields that represent cine-frame-specific dynamics. STINR-MR was evaluated using MR data simulated from the 4D extended cardiac-torso (XCAT) digital phantom, as well as two MR datasets acquired clinically from human subjects. Its reconstruction accuracy was also compared with that of the model-based non-rigid motion estimation method (MR-MOTUS) and a deep learning-based method (TEMPEST).Main results. STINR-MR can reconstruct 3D cine-MR images with high temporal (<100 ms) and spatial (3 mm) resolutions. Compared with MR-MOTUS and TEMPEST, STINR-MR consistently reconstructed images with better image quality and fewer artifacts and achieved superior tumor localization accuracy via the solved dynamic DVFs. For the XCAT study, STINR reconstructed the tumors to a mean ± SD center-of-mass error of 0.9 ± 0.4 mm, compared to 3.4 ± 1.0 mm of the MR-MOTUS method. The high-frame-rate reconstruction capability of STINR-MR allows different irregular motion patterns to be accurately captured.Significance. STINR-MR provides a lightweight and efficient framework for accurate 3D cine-MRI reconstruction. It is a 'one-shot' method that does not require external data for pre-training, allowing it to avoid generalizability issues typically encountered in deep learning-based methods.

Joint Stroke Tracing and Correspondence for 2D Animation

To alleviate human labor in redrawing keyframes with ordered vector strokes for automatic inbetweening, we for the first time propose a joint stroke tracing and correspondence approach. Given consecutive raster keyframes along with a single vector image of the starting frame as a guidance, the approach generates vector drawings for the remaining keyframes while ensuring one-to-one stroke correspondence. Our framework trained on clean line drawings generalizes to rough sketches, and the generated results can be imported into inbetweening systems to produce inbetween sequences. Hence, the method is compatible with standard 2D animation workflow. An adaptive spatial transformation module (ASTM) is introduced to handle non-rigid motions and stroke distortion. We collect a dataset for training with 10k+ pairs of raster frames and their vector drawings with stroke correspondence. Comprehensive validations on real clean and rough animated frames manifest the effectiveness of our method and superiority to existing methods.

Extended MRI-based PET motion correction for cardiac PET/MRI

PurposeA 2D image navigator (iNAV) based 3D whole-heart sequence has been used to perform MRI and PET non-rigid respiratory motion correction for hybrid PET/MRI. However, only the PET data acquired during the acquisition of the 3D whole-heart MRI is corrected for respiratory motion. This study introduces and evaluates an MRI-based respiratory motion correction method of the complete PET data.MethodsTwelve oncology patients scheduled for an additional cardiac 18F-Fluorodeoxyglucose (18F-FDG) PET/MRI and 15 patients with coronary artery disease (CAD) scheduled for cardiac 18F-Choline (18F-FCH) PET/MRI were included. A 2D iNAV recorded the respiratory motion of the myocardium during the 3D whole-heart coronary MR angiography (CMRA) acquisition (~ 10 min). A respiratory belt was used to record the respiratory motion throughout the entire PET/MRI examination (~ 30–90 min). The simultaneously acquired iNAV and respiratory belt signal were used to divide the acquired PET data into 4 bins. The binning was then extended for the complete respiratory belt signal. Data acquired at each bin was reconstructed and combined using iNAV-based motion fields to create a respiratory motion-corrected PET image. Motion-corrected (MC) and non-motion-corrected (NMC) datasets were compared. Gating was also performed to correct cardiac motion. The SUVmax and TBRmax values were calculated for the myocardial wall or a vulnerable coronary plaque for the 18F-FDG and 18F-FCH datasets, respectively.ResultsA pair-wise comparison showed that the SUVmax and TBRmax values of the motion corrected (MC) datasets were significantly higher than those for the non-motion-corrected (NMC) datasets (8.2 ± 1.0 vs 7.5 ± 1.0, p < 0.01 and 1.9 ± 0.2 vs 1.2 ± 0.2, p < 0.01, respectively). In addition, the SUVmax and TBRmax of the motion corrected and gated (MC_G) reconstructions were also higher than that of the non-motion-corrected but gated (NMC_G) datasets, although for the TBRmax this difference was not statistically significant (9.6 ± 1.3 vs 9.1 ± 1.2, p = 0.02 and 2.6 ± 0.3 vs 2.4 ± 0.3, p = 0.16, respectively). The respiratory motion-correction did not lead to a change in the signal to noise ratio.ConclusionThe proposed respiratory motion correction method for hybrid PET/MRI improved the image quality of cardiovascular PET scans by increased SUVmax and TBRmax values while maintaining the signal-to-noise ratio.Trial registration METC162043 registered 01/03/2017.

A method of applying deep learning based optical flow algorithm to river flow discharge measurement

River flow discharge monitoring is one of the critical tasks performed at hydrological stations. The large-scale particle image velocimetry (LSPIV) method widely used in hydrological stations is often limited by a lack of floating objects and has a high computational complexity. The space-time image velocimetry method is susceptible to noise interference and requires high stability of the flow over time. This paper proposes a flow measurement method based on the recurrent all-pairs field transforms for optical flow (RAFT) algorithm. The 4D correlation volume pyramid in the RAFT network structure can effectively handle changing and complex flow conditions. The convolutional block attention module is introduced into the optical flow update module after the 4D correlation volume pyramid, enhancing the ability to capture complex flow surface information. Additionally, feature extraction adds deformable convolution to expand the receptive field of the flow image, which has better adaptability in non-rigid motion. To validate the effectiveness of the new method (RAFT-D-C), this paper conducts comparative experiments with both existing and new methods. The experimental results show that RAFT-D-C has relative errors of 2.13% and 4.41% for the average flow velocity of two rivers and relative errors of 2.19% and 3.05% for the total discharge, respectively. RAFT-D-C provides improved accuracy compared to other methods and requires less computational run time than the frequently used LSPIV method.

JDECMC: Improving JDE based multi-object tracking with Camera Motion Compensation

Jointly Detection and Embedding (JDE) based approaches estimate bounding boxes and embedding features of objects with a single network in multi-object tracking. Due to the crowded scenarios or non-rigid camera motion, the bounding box location in the image can shift dramatically, which affects the performance of the trackers. In addition, JDE-based approaches fuse the target appearance information and location information by applying the same rule, which could fail to associate when the target is lost or occluded. To address these issues, we propose a simple yet effective architecture based on JDE, called JDECMC. Our approach incorporates the embedding and location distance of the object, which combines embedding cosine distance and location distance of objects, as well as Camera Motion Compensation (CMC) to predict the correct location of the bounding box. This is especially useful in scenarios where dynamic camera motion can cause the bounding box to shift significantly. By enhancing the accuracy of the tracking system, JDECMC achieves a MOTA of 74.2%, 72.0%, and 90.5% on the MOT17, MOT20 and DanceTrack benchmark datasets, respectively, while also scoring an IDF1 of 72.5%, 72.4%, and 79.8% on those benchmarks. The experimental results demonstrate our effectiveness and advantages over the other one-shot trackers. The code is available at https://github.com/Melikamuliyih/JDECMC.

Non-rigid motion-compensated 3D whole-heart T2 mapping in a hybrid 3T PET-MR system.

Simultaneous PET-MRI improves inflammatory cardiac disease diagnosis. However, challenges persist in respiratory motion and mis-registration between free-breathing 3D PET and 2D breath-held MR images. We propose a free-breathing non-rigid motion-compensated 3D T2 -mapping sequence enabling whole-heart myocardial tissue characterization in a hybrid 3T PET-MR system and provides non-rigid respiratory motion fields to correct also simultaneously acquired PET data. Free-breathing 3D whole-heart T2 -mapping was implemented on a hybrid 3T PET-MRI system. Three datasets were acquired with different T2 -preparation modules (0, 28, 55 ms) using 3-fold undersampled variable-density Cartesian trajectory. Respiratory motion was estimated via virtual 3D image navigators, enabling multi-contrast non-rigid motion-corrected MR reconstruction. T2 -maps were computed using dictionary-matching. Approach was tested in phantom, 8 healthy subjects, 14 MR only and 2 PET-MR patients with suspected cardiac disease and compared with spin echo reference (phantom) and clinical 2D T2 -mapping (in-vivo). Phantom results show a high correlation (R2 = 0.996) between proposed approach and gold standard 2D T2 mapping. In-vivo 3D T2 -mapping average values in healthy subjects (39.0 ± 1.4 ms) and patients (healthy tissue) (39.1 ± 1.4 ms) agree with conventional 2D T2 -mapping (healthy = 38.6 ± 1.2 ms, patients = 40.3 ± 1.7 ms). Bland-Altman analysis reveals bias of 1.8 ms and 95% limits of agreement (LOA) of -2.4-6 ms for healthy subjects, and bias of 1.3 ms and 95% LOA of -1.9 to 4.6 ms for patients. Validated efficient 3D whole-heart T2 -mapping at hybrid 3T PET-MRI provides myocardial inflammation characterization and non-rigid respiratory motion fields for simultaneous PET data correction. Comparable T2 values were achieved with both 3D and 2D methods. Improved image quality was observed in the PET images after MR-based motion correction.

VoxelMorph-Based Deep Learning Motion Correction for Ultrasound Localization Microscopy of Spinal Cord.

Accurate assessment of spinal cord vasculature is important for the urgent diagnosis of injury and subsequent treatment. Ultrasound localization microscopy (ULM) offers super-resolution imaging of microvasculature by localizing and tracking individual microbubbles across multiple frames. However, a long data acquisition often involves significant motion artifacts caused by breathing and heartbeat, which further impairs the resolution of ULM. This effect is particularly pronounced in spinal cord imaging due to respiratory movement. We propose a VoxelMorph-based deep learning motion correction method to enhance ULM performance in spinal cord imaging. Simulations were conducted to demonstrate the motion estimation accuracy of the proposed method, achieving a mean absolute error of 8 μm. Results from in vivo experiments show that the proposed method efficiently compensates for rigid and nonrigid motion, providing improved resolution with smaller vascular diameters and enhanced microvessel reconstruction after motion correction. Nonrigid deformation fields with varying displacement magnitudes were applied to in vivo data for assessing the robustness of the algorithm.

A Deformation Field-Based Approach for Expression Processing in Head Motion Correction

It is a routine procedure to use Iterative Closest Point (ICP) algorithm to correct motion in medical imaging of the head. However, the basic assumption of ICP is the rigid body motion, and it is not strictly valid when there is non-rigid motion (facial expression). In this study, we propose a novel method to reduce the adverse effects of facial expressions on head motion estimation. First, we design a network named DFG-Net to generate a deformation field on the RGB domain and compute a confidence map. We then extend the original ICP by embedding the confidence map representing each point’s degree of non-rigid motion. Compared to other techniques, our proposed method eliminates the negative effects of facial expression while making the most of pose information. The experiments demonstrate that our approach reduces the position error by 19.23% and orientation error by 36.37% compared to the original ICP algorithm. They also show that our method is robust to variations of expressions.