Pixel Trajectories Research Articles

Dense Continuous-Time Optical Flow From Event Cameras.

We present a method for estimating dense continuous-time optical flow from event data. Traditional dense optical flow methods compute the pixel displacement between two images. Due to missing information, these approaches cannot recover the pixel trajectories in the blind time between two images. In this work, we show that it is possible to compute per-pixel, continuous-time optical flow using events from an event camera. Events provide temporally fine-grained information about movement in pixel space due to their asynchronous nature and microsecond response time. We leverage these benefits to predict pixel trajectories densely in continuous time via parameterized Bézier curves. To achieve this, we build a neural network with strong inductive biases for this task: First, we build multiple sequential correlation volumes in time using event data. Second, we use Bézier curves to index these correlation volumes at multiple timestamps along the trajectory. Third, we use the retrieved correlation to update the Bézier curve representations iteratively. Our method can optionally include image pairs to boost performance further. To the best of our knowledge, our model is the first method that can regress dense pixel trajectories from event data. To train and evaluate our model, we introduce a synthetic dataset (MultiFlow) that features moving objects and ground truth trajectories for every pixel. Our quantitative experiments not only suggest that our method successfully predicts pixel trajectories in continuous time but also that it is competitive in the traditional two-view pixel displacement metric on MultiFlow and DSEC-Flow. Open source code and datasets are released to the public.

Lagrangian motion magnification with double sparse optical flow decomposition

Microexpressions are fast and spatially small facial expressions that are difficult to detect. Therefore, motion magnification techniques, which aim at amplifying and hence revealing subtle motion in videos, appear useful for handling such expressions. There are basically two main approaches, namely, via Eulerian or Lagrangian techniques. While the first one magnifies motion implicitly by operating directly on image pixels, the Lagrangian approach uses optical flow (OF) techniques to extract and magnify pixel trajectories. In this study, we propose a novel approach for local Lagrangian motion magnification of facial micro-motions. Our contribution is 3-fold: first, we fine tune the recurrent all-pairs field transforms (RAFT) for OFs deep learning approach for faces by adding ground truth obtained from the variational dense inverse search (DIS) for the OF algorithm applied to the CASME II video set of facial micro expressions. This enables us to produce OFs of facial videos in an efficient and sufficiently accurate way. Second, since facial micro-motions are both local in space and time, we propose to approximate the OF field by sparse components both in space and time leading to a double sparse decomposition. Third, we use this decomposition to magnify micro-motions in specific areas of the face, where we introduce a new forward warping strategy using a triangular splitting of the image grid and barycentric interpolation of the RGB vectors at the corners of the transformed triangles. We demonstrate the feasibility of our approach by various examples.

Fine-grained action segmentation based on complementary Frame-level Classification Model and Action-wise Regression Model

Over-segmentation is a critical issue in fine-grained action segmentation tasks, such as repetitive motion detection and factory construction progress monitoring. To address these problems, we segment the temporal structure of each action instance into three action phases, and propose an innovative framework. This framework consists of a spatial–temporal feature extractor and two paralleled models: the Frame-level Classification Model (FCM) and the Action-wise Regression Model (ARM). We use frames as appearance stream and pixel trajectories as motion stream. The fused features are fed to the FCM for frame-level classification and the ARM for action phase regression. At inference time, the frame-level classification sequence is fine-tuned by regression. In this way, our model reduces over-segmentation and segments actions more accurately. We test the model on three datasets: 50Salads, GTEA, and our self-built dataset in which workers try to drill pipes in a coal mine well. Experimental results show that our method is better than other methods, and at the same time, over-segmentation is significantly reduced after adding the ARM.

Functional movement screen dataset collected with two Azure Kinect depth sensors

This paper presents a dataset for vision-based autonomous Functional Movement Screen (FMS) collected from 45 human subjects of different ages (18–59 years old) executing the following movements: deep squat, hurdle step, in-line lunge, shoulder mobility, active straight raise, trunk stability push-up and rotary stability. Specifically, shoulder mobility was performed only once by different subjects, while the other movements were repeated for three episodes each. Each episode was saved as one record and was annotated from 0 to 3 by three FMS experts. The main strength of our database is twofold. One is the multimodal data provided, including color images, depth images, quaternions, 3D human skeleton joints and 2D pixel trajectories of 32 joints. The other is the multiview data collected from the two synchronized Azure Kinect sensors in front of and on the side of the subjects. Finally, our dataset contains a total of 1812 recordings, with 3624 episodes. The size of the dataset is 190 GB. This dataset provides the opportunity for automatic action quality evaluation of FMS.

(OA43) Delineation of a Cardiac PRV Using Real-Time Magnetic Resonance Imaging for Cardiac Protection in Thoracic and Breast Radiation Therapy

Historically, increased cardiac radiation dose has been associated with late cardiotoxicity events in patients with thoracic and breast malignancies. Recently, a prospective trial also revealed correlation between cardiac dose and early toxicity, including reduced overall survival in patients receiving curative-intent treatment. While breath-hold techniques reduce motion, they cannot account for cardiac cycle motion. We conducted a prospective cine magnetic resonance imaging (cMRI) study and applied an established scale-invariant feature transform (SIFT)-based algorithm to track and evaluate heart motion. We hypothesized that cMRI could be used to define a cardiac planning organ-at-risk volume (PRV) for routine application in radiation planning to improve cardiac protection. Imaging was performed on a 1.5 Tesla MR scanner (Philips Ingenia 1.5T MR System). 16 acquisitions were obtained per subject using a balanced-steady-state gradient echo sequence. Single-plane cMRI was performed in four sequential sagittal and four sequential coronal planes, spanning the heart in equidistant left-right (LR) and anterior-posterior (AP) intervals. Free-breathing (FB) and deep-inspiratory-breath-hold (DIBH) acquisitions were performed at each plane. In-plane cardiac motion was assessed on all image sets for each subject. The reference image and each image within the corresponding acquisition image set were registered via a previously published technique that describes pixels by high-dimensional descriptors, incorporating nearby intensity gradients rather than pixel intensity alone. Each descriptor, defined by the scale-invariant feature transform (SIFT) algorithm, is a set of histograms of intensity gradients within sub-regions surrounding a given pixel. Pixel trajectories were then identified by minimizing an objective function to locate best-matching descriptors between the reference image and each image within an acquisition. Subject-specific pixel motion ranges were derived by averaging slice-by-slice motion of each pixel to yield mean values for AP, LR, and superior-inferior (SI) directions. For each subject, average maximum range of pixel displacement and average pixel-based interquartile range (IQR) of displacement were calculated. These values were then utilized and averaged across the cohort to calculate planning organ-at-risk volume (PRV) expansions at FB and DIBH. 20 subjects were enrolled with median age 34.5 years (range 20-78), BMI 23.75 (range 17.7-33.3), and height 64 inches (range 57-69). A total of 3,120 image frames were collected per patient in coronal and sagittal planes, at DIBH and FB, with 62,400 frames analyzed across the cohort. SIFT methodology was successfully implemented in all patients to track motion. The cohort average of maximum motion ranges comprised PRV margin expansions of 13.7±1.8mm SI, 6.9±1.4mm AP, and 7.7±1.2mm LR plane at FB; expansions for DIBH were 7.1±1.5mm SI, 5.1±1.4mm AP, and 5.6±1.3mm LR. IQR-based PRVs, calculated to encompass the most frequent range of cardiac displacement, comprised 3.7±0.7mm SI, 1.2±0.4mm AP, and 1.4±0.3mm LR at FB and 1.8±0.6mm SI, 1.0±0.3mm AP, and 1.3±0.4mm LR at DIBH. IQR-based PRV expansions were then successfully applied for both locally-advanced lung cancer and breast cancer treatment planning. We successfully utilized SIFT-based motion-tracking for analysis of 62,400 prospectively gathered cine MR image frames to identify and apply a cardiac PRV for treatment planning to improve cardiac protection in breast and thoracic radiation.

SIFT-based dense pixel tracking on 0.35 T cine-MR images acquired during image-guided radiation therapy with application to gating optimization.

To first demonstrate the viability of applying an image processing technique for tracking regions on low-contrast cine-MR images acquired during image-guided radiation therapy, and then outline a scheme that uses tracking data for optimizing gating results in a patient-specific manner. A first-generation MR-IGRT system-treating patients since January 2014-integrates a 0.35 T MR scanner into an annular gantry consisting of three independent Co-60 sources. Obtaining adequate frame rates for capturing relevant patient motion across large fields-of-view currently requires coarse in-plane spatial resolution. This study initially (1) investigate the feasibility of rapidly tracking dense pixel correspondences across single, sagittal plane images (with both moderate signal-to-noise and spatial resolution) using a matching objective for highly descriptive vectors called scale-invariant feature transform (SIFT) descriptors associated to all pixels that describe intensity gradients in local regions around each pixel. To more accurately track features, (2) harmonic analysis was then applied to all pixel trajectories within a region-of-interest across a short training period. In particular, the procedure adjusts the motion of outlying trajectories whose relative spectral power within a frequency bandwidth consistent with respiration (or another form of periodic motion) does not exceed a threshold value that is manually specified following the training period. To evaluate the tracking reliability after applying this correction, conventional metrics-including Dice similarity coefficients (DSCs), mean tracking errors (MTEs), and Hausdorff distances (HD)-were used to compare target segmentations obtained via tracking to manually delineated segmentations. Upon confirming the viability of this descriptor-based procedure for reliably tracking features, the study (3) outlines a scheme for optimizing gating parameters-including relative target position and a tolerable margin about this position-derived from a probability density function that is constructed using tracking results obtained just prior to treatment. The feasibility of applying the matching objective for SIFT descriptors toward pixel-by-pixel tracking on cine-MR acquisitions was first retrospectively demonstrated for 19 treatments (spanning various sites). Both with and without motion correction based on harmonic analysis, sub-pixel MTEs were obtained. A mean DSC value spanning all patients of 0.916 ± 0.001 was obtained without motion correction, with DSC values exceeding 0.85 for all patients considered. While most patients show accurate tracking without motion correction, harmonic analysis does yield substantial gain in accuracy (defined using HDs) for three particularly challenging subjects. An application of tracking toward a gating optimization procedure was then demonstrated that should allow a physician to balance beam-on time and tissue sparing in a patient-specific manner by tuning several intuitive parameters. Tracking results show high fidelity in assessing intrafractional motion observed on cine-MR acquisitions. Incorporating harmonic analysis during a training period improves the robustness of the tracking for challenging targets. The concomitant gating optimization procedure should allow for physicians to quantitatively assess gating effectiveness quickly just prior to treatment in a patient-specific manner.