Spatio-temporal Image Sequence Research Articles

Visual neurons recognize complex image transformations.

Natural visual scenes are dominated by sequences of transforming images. Spatial visual information is thought to be processed by detection of elemental stimulus features which are recomposed into scenes. How image information is integrated over time is unclear. We explored visual information encoding in the optic tectum. Unbiased stimulus presentation shows that the majority of tectal neurons recognize image sequences. This is achieved by temporally dynamic response properties, which encode complex image transitions over several hundred milliseconds. Calcium imaging reveals that neurons that encode spatiotemporal image sequences fire in spike sequences that predict a logical diagram of spatiotemporal information processing. Furthermore, the temporal scale of visual information is tuned by experience. This study indicates how neurons recognize dynamic visual scenes that transform over time.

Investigating a three-dimensional convolution recognition model for acoustic emission signal analysis during uniaxial compression failure of coal

Acoustic emission predicts coal sample failure, vital for early warnings and monitoring. The study emphasizes using acoustic emission to predict coal sample failure patterns and establish a discriminative model for monitoring. It employs a lightweight 3D convolutional model combining DenseNet with Group Convolution (GC) and the "squeeze-and-excitation" (SE) module. Coal samples from Xinglongzhuang Coal Mine underwent uniaxial compression, and a subset of experiments with prominent features was selected. The resulting acoustic emission parameters were then transformed into spatiotemporal image sequences for input to the model. Successfully identifying hazardous coal damage stages, all four network structures (DenseNet, DenseNet + GC, DenseNet + SE, DenseNet + GC + SE) achieved over 98.29% predictive accuracy in verification samples. The model exhibited a recall rate of over 98.02% for high-risk samples, effectively capturing spatiotemporal information. DenseNet + GC + SE showed a probability distribution focusing on different risk levels. By integrating group convolution and SE modules, this model significantly reduced both model and time complexity while preserving precision, enhancing efficiency. It effectively discerns diverse acoustic emission features through group convolution and evaluates their significance using the SE module. Compared to the traditional C3D neural network, this model greatly reduces complexity and time requirements while improving efficiency in distinguishing between various damage stages.

TasselLFANet: a novel lightweight multi-branch feature aggregation neural network for high-throughput image-based maize tassels detection and counting.

Accurately and rapidly counting the number of maize tassels is critical for maize breeding, management, and monitoring the growth stage of maize plants. With the advent of high-throughput phenotyping platforms and the availability of large-scale datasets, there is a pressing need to automate this task for genotype and phenotype analysis. Computer vision technology has been increasingly applied in plant science, offering a promising solution for automated monitoring of a large number of plants. However, the current state-of-the-art image algorithms are hindered by hardware limitations, which compromise the balance between algorithmic capacity, running speed, and overall performance, making it difficult to apply them in real-time sensing field environments. Thus, we propose a novel lightweight neural network, named TasselLFANet, with an efficient and powerful structure for accurately and efficiently detecting and counting maize tassels in high spatiotemporal image sequences. Our proposed approach improves the feature-learning ability of TasselLFANet by adopting a cross-stage fusion strategy that balances the variability of different layers. Additionally, TasselLFANet utilizes multiple receptive fields to capture diverse feature representations, and incorporates an innovative visual channel attention module to detect and capture features more flexibly and precisely. We conducted a series of comparative experiments on a new, highly informative dataset called MrMT, which demonstrate that TasselLFANet outperforms the latest batch of lightweight networks in terms of performance, flexibility, and adaptability, achieving an F1 measure value of 94.4%, a mAP.@5 value of 96.8%, and having only 6.0M parameters. Moreover, compared with the regression-based TasselNetV3-Seg† model, our proposed model achieves superior counting performance, with a mean absolute error (MAE) of 1.80, a root mean square error (RMSE) of 2.68, and a R2 of 0.99. The proposed model meets the accuracy and speed requirements of the vision system in maize tassel detection. Furthermore, our proposed method is reliable and unaffected by geographical changes, providing essential technical support for computerized counting in the field.

Rock mass fracture maps prediction based on spatiotemporal image sequence modeling

AbstractDiscontinuities in rock mass are the characteristic challenge of rock tunnel engineering projects, which have a vital impact on rock mass exposures' mechanical and hydrological characteristics. There is a growing demand for predicting the fracture maps during tunnel excavation to ensure a smooth tunnel excavation process. The computer vision measurement of fractures in the tunnel surface is a current hot spot, but the traditional statistical analysis methods for fractures are still mainstream. This paper uses a novel perspective of the time‐space sequence to explain the continuously exposed rock mass during tunneling. A spatial‐aware recurrent neural network is proposed, which takes the historical fracture maps as the input to predict the unexcavated part. The experimental results suggest that the proposed model produces reliable performance and is superior to the other two state‐of‐the‐art deep learning models. Moreover, the test on the site rock tunnel data suggested promising results for fracture map predictions.

Spatio-Temporal Mitosis Detection in Time-Lapse Phase-Contrast Microscopy Image Sequences: A Benchmark.

In this paper, we report the results of the first international contest on mitosis detection in phase-contrast microscopy image sequences (https://www.iti-tju.org/mitosisdetection), which was held at the workshop of computer vision for microscopy image analysis (CVMI) in CVPR 2019. This contest aims to promote research on spatiotemporal mitosis detection under microscopy images. In this contest, we released a large-scale time-lapse phase-contrast microscopy image dataset (C2C12-16) for the mitosis detection task. Compared with the previous popular datasets (e.g., C2C12, C3H10), C2C12-16 contains more annotated mitotic events and more diverse cell culture environments. A total of ten different mitosis detection methods were submitted in the contest and evaluated on the test sets of four different cell culture environments in C2C12-16. In this benchmark, we describe all methods and conduct a thorough analysis based on their performances and discuss a feasible direction for mitosis detection. To the best of our knowledge, this is the first benchmark for the mitosis detection problem using a time-lapse phase-contrast microscopy spatiotemporal image sequence model.

A Deep Learning Pipeline for Identification of Motor Units in Musculoskeletal Ultrasound

Ultrasound imaging provides information from a large part of the muscle. It has recently been shown that ultrafast ultrasound imaging can be used to record and analyze the mechanical response of individual MUs using blind source separation. In this work, we present an alternative method - a deep learning pipeline - to identify active MUs in ultrasound image sequences, including segmentation of their territories and signal estimation of their mechanical responses (twitch train). We train and evaluate the model using simulated data mimicking the complex activation pattern of tens of activated MUs with overlapping territories and partially synchronized activation patterns. Using a slow fusion approach (based on 3D CNNs), we transform the spatiotemporal image sequence data to 2D representations and apply a deep neural network architecture for segmentation. Next, we employ a second deep neural network architecture for signal estimation. The results show that the proposed pipeline can effectively identify individual MUs, estimate their territories, and estimate their twitch train signal at low contraction forces. The framework can retain spatio-temporal consistencies and information of the mechanical response of MU activity even when the ultrasound image sequences are transformed into a 2D representation for compatibility with more traditional computer vision and image processing techniques. The proposed pipeline is potentially useful to identify simultaneously active MUs in whole muscles in ultrasound image sequences of voluntary skeletal muscle contractions at low force levels.

Lightness perception for surfaces moving through different illumination levels.

Lightness perception has mainly been studied with static scenes so far. This study presents four experiments investigating lightness perception under dynamic illumination conditions. We asked participants for lightness matches of a virtual three-dimensional target moving through a light field while their eye movements were recorded. We found that the target appeared differently, depending on the direction of motion in the light field and its precise position in the light field. Lightness was also strongly affected by the choice of fixation positions with the spatiotemporal image sequence. Overall, lightness constancy was improved when observers could freely view the object, over when they were forced to fixate certain regions. Our results show that dynamic scenes and nonuniform light fields are particularly challenging for our visual system. Eye movements in such scenarios are chosen to improve lightness constancy.

Model-based analysis of cerebrovascular diseases combining 3D and 4D MRA datasets

Abstract The cerebral stroke is a major cause for death and disability. Clinical diagnosis, therapy, and research of stroke can considerably benefit from modern image acquisition methods, which enable a detailed analysis of cerebral blood vessel anatomy as well as an examination of macrovascular and tissue blood flow dynamics. However, visual screening of these datasets can be complex and time-consuming due to the vast amount of data. This article provides an overview of a dissertation, which addresses the problem of an automatic combined analysis and visualization of high-resolution 3D and spatiotemporal (4D) image sequences from the same patient to support diagnosis, treatment decision, and research of cerebrovascular diseases. Therefore, automatic methods for the cerebrovascular segmentation, analysis of the cerebral blood flow and tissue perfusion, as well as the combined quantitative analysis and visualization of the vessel morphology and blood flow dynamics were developed. Apart from a potential clinical application, the developed methods have already proven useful in multiple clinical research studies.

Multiple-class spatiotemporal flow estimation using a modified neural gas algorithm

The problem of estimating optical flow fields corresponding to multiple moving objects in a spatiotemporal image sequence is addressed. A modified version of the neural gas (NG) unsupervised learning algorithm is used to implement a nonlinear interpolation strategy to overcome the aperture problem encountered during local motion estimation. Local motion constraints are formulated, and the best information over four point pairs is used to produce a single motion estimate. Wherever the aperture problem is encountered the minimum-norm estimate is produced. These local estimates are then refined using the modified NG. NG provides a framework for the fusion of local incomplete motion information into complete global estimates. Due to the self-organizing nature of NG the number of motion classes need not be specified a priori. The technique leads to generation of an optical flow field without the smearing of flow fields encountered in regularization-based techniques. Motion estimation results obtained on synthetic natural image sequences are shown.

VV: a viewer for the evaluation of 4D image registration

We propose an open source and cross platform medical image viewer, named VV, designed for qualita tive evaluation of images registration, in particular for deformable registration of 4D CT images. VV can display multiple spatio-temporal image sequences (2D+t or 3D+t) and contains several tools for comparing images using transparency or fusion, for visualizing deformation fields, for defining landmarks. It is used in the field of radiation therapy to help researchers and clinicians evaluate deformation in 4D CT images of the thorax. It is implemented in C++, with ITK, VTK and QT open source version, runs on Linux and windows and is freely available to the community.

Motion Artifact Reducing Reconstruction of 4D CT Image Data for the Analysis of Respiratory Dynamics

Respiratory motion represents a major problem in radiotherapy of thoracic and abdominal tumors. Methods for compensation require comprehensive knowledge of underlying dynamics. Therefore, 4D (= 3D + t) CT data can be helpful. But modern CT scanners cannot scan a large region of interest simultaneously. So patients have to be scanned in segments. Commonly used approaches for reconstructing the data segments into 4D CT images cause motion artifacts. In order to reduce the artifacts, a new method for 4D CT reconstruction is presented. The resulting data sets are used to analyze respiratory motion. Spatiotemporal CT image sequences of lung cancer patients were acquired using a multi-slice CT in cine mode during free breathing. 4D CT reconstruction was done by optical flow based temporal interpolation. The resulting 4D image data were compared with data generated by the commonly used nearest neighbor reconstruction. Subsequent motion analysis is mainly concerned with tumor mobility. The presented optical flow-based method enables the reconstruction of 3D CT images at arbitrarily chosen points of the patient's breathing cycle. A considerable reduction of motion artifacts has been proven in eight patient data sets. Motion analysis showed that tumor mobility differs strongly between the patients. Due to the proved reduction of motion artifacts, the optical flow-based 4D CT reconstruction offers the possibility of high-quality motion analysis. Because the method is based on an interpolation scheme, it additionally has the potential to enable the reconstruction of 4D CT data from a lesser number of scans.

Noise estimation for video processing based on spatio-temporal gradients

We propose an efficient and accurate wavelet-based noise estimation method for white Gaussian noise in video sequences. The proposed method analyzes the distribution of spatial and temporal gradients in the video sequence in order to estimate the noise variance. The estimate is derived from the most frequent gradient in the two distributions and is compensated for the errors due to the spatio-temporal image sequence content, by a novel correction function. The spatial and temporal gradients are determined from the finest scale of the spatial and temporal wavelet transform, respectively. The main application of the noise estimation algorithm is in wavelet-based video processing. The results show that the proposed method is more accurate than other state-of-the-art noise estimation techniques and less sensitive to varying spatio-temporal content and noise level.

3-D IIR filtering using decimated DFT-polyphase filter bank structures

Two three-dimensional (3-D) under-decimated uniform discrete Fourier transform polyphase filter bank structures are proposed along with two applications: first, the sub-pixel motion discrimination of two-dimensional spatial objects moving with approximately constant local velocity in a noisy 3-D spatio-temporal image sequence and, second, the selective filtering of 3-D spatio-temporal broad-band plane waves based on their directions of arrival. The desired 3-D filter passband shapes are realized utilizing combinations of highly selective first-order 3-D infinite-impulse response frequency-planar filters in each band between the analysis and synthesis sections. Measured spatio-temporal performance confirms the high-quality broad-band transmission of passband signals, high directional selectivity and low computational complexity.

Spatiotemporal anisotropic diffusion filtering to improve signal-to-noise ratios and object restoration in fluorescence microscopic image sequences.

We present an approach for significantly improving the quantitative analysis of motion in noisy fluorescence microscopic image sequences. The new partial differential equation based method is a general extension of a 2-D nonlinear anisotropic diffusion filtering scheme to a specially adapted 3-D nonlinear anisotropic diffusion filtering scheme, with two spatial image dimensions and the time t in the image sequence as the third dimension. Motion in image sequences is considered as oriented, line-like structures in the spatiotemporal x,y,t domain, which are determined by the structure tensor method. Image enhancement is achieved by a structure adopted smoothing kernel in three dimensions, thereby using the full 3-D information inherent in spatiotemporal image sequences. As an example for low signal-to-noise ratio (SNR) microscopic image sequences we have applied this method to noisy in vitro motility assay data, where fluorescently labeled actin filaments move over a surface of immobilized myosin. With the 3-D anisotropic diffusion filtering the SNR is significantly improved (by a factor of 3.8) and closed object structures are reliably restored, which were originally degraded by noise. Generally, this approach is very valuable for all applications where motion has to be measured quantitatively in low light level fluorescence microscopic image sequences of cellular, subcellular, and molecular processes.

Visual understanding of dynamic hand gestures

Analysis of a dynamic hand gesture requires processing a spatio-temporal image sequence. The actual length of the sequence varies with each instantiation of the gesture. The key idea behind solving the problem is to translate the richness of the human gestural communication power to a machine for a better man–machine interaction. We propose a novel vision-based system for automatic interpretation of a limited set of dynamic hand gestures. This involves extracting the temporal signature of the hand motion from the performed gesture. The concept of motion energy is used to estimate the dominant motion from an image sequence. To achieve the desired result, we introduce the concept of modeling the dynamic hand gesture using a finite state machine. The temporal signature is subsequently analyzed by the finite state machine to interpret automatically the performed gesture.

3‐D spatiotemporal subband decomposition of image sequences by mathematical morphology

Abstract An efficient 3‐D spatiotemporal image sequence decomposition method using mathematical morphology is described in this paper. The method can be used to decompose the spectrum of the input signal into 8 and 4 spatiotemporal subband images. It does this using two different sets of structuring elements. After decomposition, each band image can be decimated and coded for data transmission. This subband pyramid scheme preserves the number of pixels which existed in the original image, has an efficient hierarchical data structure, and allows parallel implementation. Therefore, this scheme has great potential for High Definition Television (HDTV) coding, multimedium video compression, etc. As regards filtering, the unique advantages of morphology over the linear filtering approach are: 1) it utilizes direct geometric interpretations; and 2) it is simple and efficient when used in conjunction with parallel/pipelining hardware. Some image sequence examples are given to show the effectiveness of this approach.

Pel-recursive motion field estimation from image sequences

In this paper, a general pel-recursive motion field estimator is presented. The algorithm contains a motion-compensated spatiotemporal motion field vector predictor, an adaptive vector update that is based on a motion-compensated spatiotemporal image sequence model, and an estimation control strategy. The tuning is discussed in detail, and trade-offs in the estimator design, relations with earlier proposed estimators, and our specific choices are outlined. The extensions contained in the algorithm are stepwise shown to improve specific characteristics of the estimator such as its robustness, speed of convergence, and estimation accuracy. The general algorithm is shown to outperform its degenerated versions by both visual and objective comparisons of the estimation results.

Motion displacement estimation using an affine model for image matching

A model is developed for estimating the displacement field in spatio-temporal image sequences that allows for affine shape deformations of corresponding spatial regions and for affine transformations of the image intensity range. This model includes the block matching method as a special case. The model parameters are found by using a least-squares algorithm. We demonstrate experimentally that the affine matching algorithm performs better in estimating displacements than other standard approaches, especially for long-range motion with possible changes in scene illumination. The algorithm is successfully applied to various classes of moving imagery, including the tracking of cloud motion.