Temporal Superpixels Research Articles

Superpixels for image and video processing based on proximity-weighted patch matching

In this paper, a temporal superpixel algorithm using proximity-weighted patch matching (PPM) is proposed to yield temporally consistent superpixels for image and video processing. PPM estimates the motion vector of a superpixel robustly, by considering the patch matching distances of neighboring superpixels as well as the superpixel itself. In each frame, we initialize superpixels by transferring the superpixel labels of the previous frame using PPM motion vectors. Then, we update the superpixel labels of boundary pixels by minimizing a cost function, which is composed of feature distance, compactness, contour, and temporal consistency terms. Finally, we carry out superpixel splitting, merging, and relabeling to regularize superpixel sizes and correct inaccurate labels. Extensive experimental results confirm that the proposed algorithm outperforms the state-of-the-art conventional algorithms significantly. Also, it is demonstrated that the proposed algorithm can be applied to video object segmentation and video saliency detection tasks.

Online Tensor Sparsifying Transform based on Temporal Superpixels from Compressive Spectral Video Measurements.

Spectral videos contain highly redundant information across spatial, spectral and temporal axes which can be exploited through a temporal-data-learned sparsifying basis. However, in compressive spectral video acquisition, tackling dictionary learning is time-consuming since it increases the computational complexity and presents drawbacks for realtime processing, where offline learning is required. This paper introduces a tensor-decomposition learning (TenDL) framework for simultaneous online sparsifying and recovering the spatialspectral- temporal information of a spectral video performed on several temporal superpixels (TSP-TenDL) for time processing reduction. The framework is composed of two main stages: preprocessing and joint estimation. The preprocessing stage includes a strategy for a grayscale approximation of the video to provide a suitable initialization of the sparsifying basis to be learned. To fully exploit the high signal correlation, a set of temporal superpixels is estimated from the grayscale approximation, reducing the reconstruction time of the large-scale data. Then, the outcome of the first stage is used to estimate the basis and the signal coefficients, where an optimization problem is solved to learn and reconstruct the basis and the signal, respectively, following a block-descent coordinate strategy. The proposed approach is compared from simulations with an offline-learned based method, traditional matrix-based recovery algorithms and the tensor-based recovery, the two latter using a fixed basis, where TSP-TenDL exhibits higher image quality results and lower computation time. Specifically, our methodology gains up to 7dB in terms of PSNR and a speedup of up to 6.6× compared with state-of-the-art counterparts.

Successional stages and their evolution in tropical forests using multi-temporal photogrammetric surface models and superpixels

Airborne photogrammetric image archives offer interesting possibilities for multi-temporal analyses of environmental evolution. The objective of this investigation was to develop a technique for classifying forest successional stages and performing multi-temporal analyses of the tree canopy based on tree height variances calculated from digital surface models (DSMs) created from photogrammetric imagery. Furthermore, our objective was to evaluate the usability of the technique in assessing the evolution of successional stages in a tropical forest. The local variance calculation in 3D space resulted in an image that was subdivided with a segmentation technique to generate small areas called superpixels. These superpixels, which use the local mean variance as an attribute, are assessed via cluster analysis to evaluate statistical similarity and define successional stage classes. The same superpixel shapes were located in georeferenced historical datasets to enable multi-temporal analysis. The cluster analysis of temporal superpixels enabled the spatiotemporal classification of forest canopy evolution. The technique was used to assess a tropical forest remnant in Brazil. Dense DSMs were generated with stereo-photogrammetric techniques using optical images (both film and digital images) from which height variances were computed. A cluster analysis of superpixels was performed to classify the forest canopy into four successional stages, which were consistent with Brazilian classification rules. The multi-temporal analysis identified six classes of forest cover evolution. Field data were collected in forest plots to validate the generated forest canopy classifications. The results showed that the proposed approach was feasible for forest cover classification and for identifying changes in the vertical forest structure and cover over time using only optical images.

The Fovea-BMO Axis Angle and Macular Thickness Vertical Asymmetry Across The Temporal Raphe.

To test the hypothesis that the fovea-Bruch's membrane opening (FoBMO) axis angle influences the thickness symmetry of the macular ganglion cell/inner plexiform layer (GCIPL) across the temporal horizontal meridian in normal subjects. Cross-sectional diagnostic study at a tertiary academic center. One hundred sixteen eyes of 60 normal subjects aged 40 to 85 years underwent spectral domain optical coherence tomography(SD-OCT) imaging. The FoBMO angle was estimated on en face infrared SD-OCT images. Posterior Pole algorithm images acquired with Spectralis SD-OCT were used to define vertical asymmetry as follows. The average thickness difference between the 3 most temporal superpixels above and below the horizontal meridian, the second row of superpixels from the horizontal meridian, and 3 central superpixels above and below the horizontal meridian were calculated. Factors influencing GCIPL thickness asymmetry were explored and changes in thickness asymmetry as a function of FoBMO angle were investigated. No demographic or clinical factors affected temporal GCIPL asymmetry (P>0.05 for all). A more (negatively) tilted FoBMO angle was associated with relatively thinner inferior compared with superior GCIPL thickness in superpixels immediately adjacent to the temporal raphe (P<0.001). The second row of temporal superpixels from the horizontal meridian (P=0.349) or central superpixels (P=0.292) did not show this tendency. Vertical GCIPL symmetry across the horizontal meridian is influenced by the FoBMO angle. SD-OCT algorithms using vertical asymmetry as a diagnostic index should be adjusted for the FoBMO angle.

Spatiotemporal Saliency Estimation by Spectral Foreground Detection

We present a novel approach for spatiotemporal saliency detection by optimizing a unified criterion of color contrast, motion contrast, appearance, and background cues. To this end, we first abstract the video by temporal superpixels. Second, we propose a novel graph structure exploiting the saliency cues to assign the edge weights. The salient segments are then extracted by applying a spectral foreground detection method, quantum cuts, on this graph. We evaluate our approach on several public datasets for video saliency and activity localization to demonstrate the favorable performance of the proposed video quantum cuts compared to the state of the art.

Unsupervised skin tissue segmentation for remote photoplethysmography

Segmentation is a critical step for many algorithms, especially for remote photoplethysmography (rPPG) applications as only the skin surface provides information. Moreover, it has been shown that the rPPG signal is not distributed homogeneously across the skin. Most of the time, algorithms get input information from face detection provided by a supervised learning of physical appearance and skin pixel selection. However, both methods show several limitations. In this paper, we propose a simple approach to implicitly select skin tissues based on their distinct pulsatility feature. The input video frames are decomposed into several temporal superpixels from which the pulse signals are extracted. A pulsatility measure from each temporal superpixel is then used to merge the pulse traces and estimate the photoplethysmogram signal. Since the most pulsatile signals provide high quality information, areas where the information is predominant are favored. We evaluated our contribution using a new publicly available dataset dedicated to rPPG algorithms comparison. The results of our experiments show that our method outperforms state of the art algorithms, without any critical face or skin detection.

Temporally Consistent Depth Map Prediction Using Deep Convolutional Neural Network and Spatial-Temporal Conditional Random Field

Deep convolutional neural networks (DCNNs) based methods recently keep setting new records on the tasks of predicting depth maps from monocular images. When dealing with video-based applications such as 2D (2-dimensional) to 3D (3-dimensional) video conversion, however, these approaches tend to produce temporally inconsistent depth maps, since their CNN models are optimized over single frames. In this paper, we address this problem by introducing a novel spatial-temporal conditional random fields (CRF) model into the DCNN architecture, which is able to enforce temporal consistency between depth map estimations over consecutive video frames. In our approach, temporally consistent superpixel (TSP) is first applied to an image sequence to establish the correspondence of targets in consecutive frames. A DCNN is then used to regress the depth value of each temporal superpixel, followed by a spatial-temporal CRF layer to model the relationship of the estimated depths in both spatial and temporal domains. The parameters in both DCNN and CRF models are jointly optimized with back propagation. Experimental results show that our approach not only is able to significantly enhance the temporal consistency of estimated depth maps over existing single-frame-based approaches, but also improves the depth estimation accuracy in terms of various evaluation metrics.

Exemplar-Based Image and Video Stylization Using Fully Convolutional Semantic Features.

Color and tone stylization in images and videos strives to enhance unique themes with artistic color and tone adjustments. It has a broad range of applications from professional image postprocessing to photo sharing over social networks. Mainstream photo enhancement softwares, such as Adobe Lightroom and Instagram, provide users with predefined styles, which are often hand-crafted through a trial-and-error process. Such photo adjustment tools lack a semantic understanding of image contents and the resulting global color transform limits the range of artistic styles it can represent. On the other hand, stylistic enhancement needs to apply distinct adjustments to various semantic regions. Such an ability enables a broader range of visual styles. In this paper, we first propose a novel deep learning architecture for exemplar-based image stylization, which learns local enhancement styles from image pairs. Our deep learning architecture consists of fully convolutional networks for automatic semantics-aware feature extraction and fully connected neural layers for adjustment prediction. Image stylization can be efficiently accomplished with a single forward pass through our deep network. To extend our deep network from image stylization to video stylization, we exploit temporal superpixels to facilitate the transfer of artistic styles from image exemplars to videos. Experiments on a number of data sets for image stylization as well as a diverse set of video clips demonstrate the effectiveness of our deep learning architecture.

Left/right hand segmentation in egocentric videos

Wearable cameras allow people to record their daily activities from a user-centered (First Person Vision) perspective. Due to their favorable location, wearable cameras frequently capture the hands of the user, and may thus represent a promising user-machine interaction tool for different applications. Existent First Person Vision methods handle hand segmentation as a background-foreground problem, ignoring two important facts: i) hands are not a single “skin-like” moving element, but a pair of interacting cooperative entities, ii) close hand interactions may lead to hand-to-hand occlusions and, as a consequence, create a single hand-like segment. These facts complicate a proper understanding of hand movements and interactions. Our approach extends traditional background-foreground strategies, by including a hand-identification step (left-right) based on a Maxwell distribution of angle and position. Hand-to-hand occlusions are addressed by exploiting temporal superpixels. The experimental results show that, in addition to a reliable left/right hand-segmentation, our approach considerably improves the traditional background-foreground hand-segmentation.

LIBSVX: A Supervoxel Library and Benchmark for Early Video Processing

Supervoxel segmentation has strong potential to be incorporated into early video analysis as superpixel segmentation has in image analysis. However, there are many plausible supervoxel methods and little understanding as to when and where each is most appropriate. Indeed, we are not aware of a single comparative study on supervoxel segmentation. To that end, we study seven supervoxel algorithms, including both off-line and streaming methods, in the context of what we consider to be a good supervoxel: namely, spatiotemporal uniformity, object/region boundary detection, region compression and parsimony. For the evaluation we propose a comprehensive suite of seven quality metrics to measure these desirable supervoxel characteristics. In addition, we evaluate the methods in a supervoxel classification task as a proxy for subsequent high-level uses of the supervoxels in video analysis. We use six existing benchmark video datasets with a variety of content-types and dense human annotations. Our findings have led us to conclusive evidence that the hierarchical graph-based (GBH), segmentation by weighted aggregation (SWA) and temporal superpixels (TSP) methods are the top-performers among the seven methods. They all perform well in terms of segmentation accuracy, but vary in regard to the other desiderata: GBH captures object boundaries best; SWA has the best potential for region compression; and TSP achieves the best undersegmentation error.

Video Object Co-Segmentation via Subspace Clustering and Quadratic Pseudo-Boolean Optimization in an MRF Framework

Multiple videos may share a common foreground object, for instance a family member in home videos, or a leading role in various clips of a movie or TV series. In this paper, we present a novel method for co-segmenting the common foreground object from a group of video sequences. The issue was seldom touched on in the literature. Starting from over-segmentation of each video into Temporal Superpixels (TSPs), we first propose a new subspace clustering algorithm which segments the videos into consistent spatio-temporal regions with multiple classes, such that the common foreground has consistent labels across different videos. The subspace clustering algorithm exploits the fact that across different videos the common foreground shares similar appearance features, while motions can be used to better differentiate regions within each video, making accurate extraction of object boundaries easier. We further formulate video object co-segmentation as a Markov Random Field (MRF) model which imposes the constraint of foreground model automatically computed or specified with little user effort. The Quadratic Pseudo-Boolean Optimization (QPBO) is used to generate the results. Experiments show that this video co-segmentation framework can achieve good quality foreground extraction results without user interaction for those videos with unrelated background, and with only moderate user interaction for those videos with similar background. Comparisons with previous work also show the superiority of our approach.