Video Representation Research Articles

Deep4D: A Compact Generative Representation for Volumetric Video

This paper introduces Deep4D a compact generative representation of shape and appearance from captured 4D volumetric video sequences of people. 4D volumetric video achieves highly realistic reproduction, replay and free-viewpoint rendering of actor performance from multiple view video acquisition systems. A deep generative network is trained on 4D video sequences of an actor performing multiple motions to learn a generative model of the dynamic shape and appearance. We demonstrate the proposed generative model can provide a compact encoded representation capable of high-quality synthesis of 4D volumetric video with two orders of magnitude compression. A variational encoder-decoder network is employed to learn an encoded latent space that maps from 3D skeletal pose to 4D shape and appearance. This enables high-quality 4D volumetric video synthesis to be driven by skeletal motion, including skeletal motion capture data. This encoded latent space supports the representation of multiple sequences with dynamic interpolation to transition between motions. Therefore we introduce Deep4D motion graphs, a direct application of the proposed generative representation. Deep4D motion graphs allow real-tiome interactive character animation whilst preserving the plausible realism of movement and appearance from the captured volumetric video. Deep4D motion graphs implicitly combine multiple captured motions from a unified representation for character animation from volumetric video, allowing novel character movements to be generated with dynamic shape and appearance detail.

Physical Representation Learning and Parameter Identification from Video Using Differentiable Physics

Representation learning for video is increasingly gaining attention in the field of computer vision. For instance, video prediction models enable activity and scene forecasting or vision-based planning and control. In this article, we investigate the combination of differentiable physics and spatial transformers in a deep action conditional video representation network. By this combination our model learns a physically interpretable latent representation and can identify physical parameters. We propose supervised and self-supervised learning methods for our architecture. In experiments, we consider simulated scenarios with pushing, sliding and colliding objects, for which we also analyze the observability of the physical properties. We demonstrate that our network can learn to encode images and identify physical properties like mass and friction from videos and action sequences. We evaluate the accuracy of our training methods, and demonstrate the ability of our method to predict future video frames from input images and actions.

What-Where-When Attention Network for video-based person re-identification

Video-based person re-identification plays a critical role in intelligent video surveillance by learning temporal correlations from consecutive video frames. Most existing methods aim to solve the challenging variations of pose, occlusion, backgrounds and so on by using attention mechanism. They almost all draw attention to the occlusion and learn occlusion-invariant video representations by abandoning the occluded area or frames, while the other areas in these frames contain sufficient spatial information and temporal cues. To overcome these drawbacks, this paper proposes a comprehensive attention mechanism covering what, where, and when to pay attention in the discriminative spatial-temporal feature learning, namely What-Where-When Attention Network (W3AN). Concretely, W3AN designs a spatial attention module to focus on pedestrian identity and obvious attributes by the importance estimating layer (What and Where), and a temporal attention module to calculate the frame-level importance (when), which is embedded into a graph attention network to exploit temporal attention features rather than computing weighted average feature for video frames like existing methods. Moreover, the experiments on three widely-recognized datasets demonstrate the effectiveness of our proposed W3AN model and the discussion of major modules elaborates the contributions of this paper.

Partitioning variability in animal behavioral videos using semi-supervised variational autoencoders.

Recent neuroscience studies demonstrate that a deeper understanding of brain function requires a deeper understanding of behavior. Detailed behavioral measurements are now often collected using video cameras, resulting in an increased need for computer vision algorithms that extract useful information from video data. Here we introduce a new video analysis tool that combines the output of supervised pose estimation algorithms (e.g. DeepLabCut) with unsupervised dimensionality reduction methods to produce interpretable, low-dimensional representations of behavioral videos that extract more information than pose estimates alone. We demonstrate this tool by extracting interpretable behavioral features from videos of three different head-fixed mouse preparations, as well as a freely moving mouse in an open field arena, and show how these interpretable features can facilitate downstream behavioral and neural analyses. We also show how the behavioral features produced by our model improve the precision and interpretation of these downstream analyses compared to using the outputs of either fully supervised or fully unsupervised methods alone.

Energy and Quality Aware Multi-Objective Resource Allocation Algorithm in Cloud

Cloud gaming has become the new service provisioning prototype that hosts the video games in the cloud and broadcasts the interactive game streaming to the players through the Internet. Here, the cloud must use massive resources for video representation and its streaming when several simultaneous players reach a particular point. Alternatively, various players may have separate necessities on Quality-of Experience, like low delay, high-video quality, etc. The challenging task is providing better service by the fixed cloud resource. Hence, there is a necessity for an energy-aware multi-resource allocation in the cloud. This paper devises a Fractional Rider-Harmony search algorithm (Fractional Rider-HSA) for resource allocation in the cloud. The Fractional Rider-HSA combines fractional calculus, Rider Optimization algorithm (ROA), and HSA. Moreover, the fitness function, like mean opinion score (MOS), gaming experience loss, fairness, energy consumption, and network parameters, is considered to determine the optimal resource allocation. The proposed model produces the maximal MOS of 0.8961, maximal gaming experience loss (QE) of 0.998, maximal fairness of 0.9991, the minimum energy consumption of 0.3109, and minimal delay 0.2266, respectively.

Video representation learning by identifying spatio-temporal transformations

Video representation learning by identifying spatio-temporal transformations

Bidirectional LSTM with saliency-aware 3D-CNN features for human action recognition

Deep convolutional neural network (DCNN) and recurrent neural network (RNN) have been proved as an imperious research area in multimedia understanding and obtained remarkable action recognition performance. However, videos contain rich motion information with varying dimensions. Existing recurrent based pipelines fail to capture long-term motion dynamics in videos with various motion scales and complex actions performed by multiple actors. Consideration of contextual and salient features is more important than mapping a video frame into a static video representation. This research work provides a novel pipeline by analyzing and processing the video information using a 3D convolution (C3D) network and newly introduced deep bidirectional LSTM. Like popular two-stream convent, we also introduce a two-stream framework with one modification; that is, we replace the optical flow stream by saliency-aware stream to avoid the computational complexity. First, we generate a saliency-aware video stream by applying the saliency-aware method. Secondly, a two-stream 3D-convolutional network (C3D) is utilized with two different types of streams, i.e., RGB stream and saliency-aware video stream, to collect both spatial and semantic temporal features. Next, a deep bidirectional LSTM network is used to learn sequential deep temporal dynamics. Finally, time-series-pooling-layer and softmax-layers classify human activity and behavior. The introduced system can learn long-term temporal dependencies and can predict complex human actions. Experimental results demonstrate the significant improvement in action recognition accuracy on different benchmark datasets.

Sign language recognition based on global-local attention

Video-level sign language recognition is still a challenging task due to the influence of sign language-independent factors and timing requirements. This paper constructs a sign language recognition framework based on global-local feature description, and proposes a three-dimensional residual global network model with attention layer and a local network model based on target detection. The global feature description is based on the whole video behavior for time series modeling. The improved timing conversion layer is used to explore the timing information of different periods and learn the video representations of different timings. In the local module the hand is located through the target detection network to highlight its key role in the whole sign language behavior, which strengthens the category differences, and compensates the global network. Experiments on two well-known Chinese sign language datasets (SLR_Dataset and DEVSIGN_D) show that the proposed method can obtain higher recognition accuracy (respectively 89.2%, 91%) and better generalization performance.

Deep-HR: Fast heart rate estimation from face video under realistic conditions

This paper presents a novel method for remote heart rate (HR) estimation. Recent studies have proved that blood pumping by the heart is highly correlated to the intense color of face pixels, and surprisingly can be utilized for remote HR estimation. Researchers successfully proposed several methods for this task, but making it work in realistic situations is still a challenging problem in computer vision community. Furthermore, learning to solve such a complex task on a dataset with very limited annotated samples is not reasonable. Consequently, researchers do not prefer to use the deep learning approaches for this problem. In this paper, we propose a simple yet efficient approach to benefit the advantages of the Deep Neural Network (DNN) by simplifying HR estimation from a complex task to learning from very correlated representation to HR. Inspired by previous work, we learn a component called Front-End (FE) to provide a discriminative representation of face videos, afterward a light deep regression auto-encoder as Back-End (BE) is learned to map the FE representation to HR. Regression task on the informative representation is simple and could be learned efficiently on limited training samples. Beside of this, to be more accurate and work well on low-quality videos, two deep encoder–decoder networks are trained to refine the output of FE. We also introduce a challenging dataset (HR-D) to show that our method can efficiently work in realistic conditions. Experimental results on HR-D and MAHNOB datasets confirm that our method could run as a real-time method and estimate the average HR better than state-of-the-art ones.

Graph-based structural difference analysis for video summarization

Keyframe extraction is an effective way to achieve video summarization. More recent studies using deep learning networks are heavily dependent on massive historical datasets for training. For practicality in real applications, we focus more on unsupervised online analysis and present a novel graph-based structural difference analysis method for this purpose. Unlike traditional methods of video representation based on raw features, undirected weighted graphs are constructed from the resulting features to represent video frames. The detailed structural changes between graphs are more consistent with the actual changes between video frames than raw features, thus making the newly proposed method robust for detecting various types of shot transitions, such as hard cuts, dissolves, wipes, and fade-ins/fade-outs. Then, considering the local influence between successive frames, a structural difference analysis of graphs is performed to detect the video shot boundaries. Finally, the median graph of each shot is obtained to extract the corresponding keyframe. Extensive experiments are conducted on three video summarization benchmark datasets. Quantitative and qualitative comparisons are made between the proposed method and other state-of-the-art methods, with the proposed method yielding remarkable improvements from 1.9% to 3.1% in terms of the F-score on the three datasets.

Dynamic-boosting attention for self-supervised video representation learning

Dynamic-boosting attention for self-supervised video representation learning

Motion cues increase focused attention towards purely visual stimuli in a nocturnal primate and drive stimulus interaction and approach/avoidance in a context-dependent manner.

Visual information is of pivotal ecological importance to monkeys, apes, and humans, whereas its role in nocturnal primate ecology is less well understood. We explored how purely visual information modulates the behavior of a nocturnal primate. Abstract (shape), photographic (shape + detail), or video (shape + detail + motion) representations of arthropod prey (Zophobas morio; food context) or a male conspecific (social context) were systematically presented to 22 individuals of the gray mouse lemur (Microcebus murinus) using a touchscreen. We assessed stimulus-directed touch interactions, durations of focused visual attention towards the different stimuli, and durations spent in the half of the setup-chamber more distant to the touchscreen (as quantification of approach/avoidance). Focused attention towards the stimulus generally increased from abstract and photographic to videographic stimuli. For the food context, indications for a parallel increase in stimulus-directed touch interactions from abstract stimulus to video were found. Approach/avoidance was independent of the stimulus type within both contexts. A comparison between the contexts under the video condition revealed higher durations of visual attention and lower stimulus avoidance in the food context compared to the social context. The number of touch interactions with the video stimulus was not generally context-dependent, but context-dependency related to sex: In the food context, animals with high and low numbers of touch interactions were equally distributed across sexes. In the social context, females showed the highest numbers of touch interactions. Numbers in males declined compared to the food context. Our results demonstrate for the first time that purely visual information modulates mouse lemur behavior and focused attention in a content- and context-specific manner, suggesting that vision is of high importance for the ecology of these nocturnal primates. The findings emphasize the need for further vision-based experiments to gain deeper insight into the evolution of visual information processing and cognition in nocturnal primates.

LANGUAGE OF EVALUATION ON ATTITUDINAL SYSTEM OF LOMBOK TOURISM PROMOTION

The study is aimed at explaining the representation of Lombok tourism promotional video from YouTube official of Indonesia tourism. It focuses on the use of verbal language. The study is descriptive-qualitative method. The source of data of this study is promotional video entitled Wonderful Indonesia Lombok available at https://www.youtube.com/watch?v=BbkFE_K_t0c. This study used the theory of attitudinal system in appraisal language proposed by Martin and White (2005) to observe the verbal language of evaluation. The result of the study shows that Indonesia tourism promotional video of Lombok emphasizes more on the using of positive attitudinal systems in appraisal language to attract the visitors and highlight Lombok destinations such as nature, culture, and the product of local people. Positive evaluation on appreciation and affect on the attractions expressed the worthiness of the resource to be visited and enjoyed with. There is an absence of judgment evaluation that deals with attitudes to people and the way they behave their character All of the positive evaluation resources merge to be one of the representations of Lombok Island as one of tourism destinations in Indonesia worth to visit.

Translation of c-Met Targeted Image-Guided Surgery Solutions in Oral Cavity Cancer-Initial Proof of Concept Data.

Simple SummaryTranslation of tumor-specific fluorescent tracers is crucial in the realization intraoperative of tumor identification during fluorescence-guided surgery. Ex vivo assessment of surgical specimens after topical tracer application has the potential to reveal the suitability of a potential surgical target prior to in vivo use in patients. In this study, the c-Met receptor was identified as a possible candidate for fluorescence-guided surgery in oral cavity cancer. Freshly excised tumor specimens obtained from ten patients with squamous cell carcinoma of the tongue were incubated with EMI-137 and imaged with a clinical-grade Cy5 prototype fluorescence camera. In total, 9/10 tumors were fluorescently illuminated, while non-visualization could be linked to non-superficial tumor localization. Immunohistochemistry revealed c-Met expression in all ten specimens. Tumor assessment was improved via video representation of the tumor-to-background ratio.Intraoperative tumor identification (extension/margins/metastases) via receptor-specific targeting is one of the ultimate promises of fluorescence-guided surgery. The translation of fluorescent tracers that enable tumor visualization forms a critical component in the realization of this approach. Ex vivo assessment of surgical specimens after topical tracer application could help provide an intermediate step between preclinical evaluation and first-in-human trials. Here, the suitability of the c-Met receptor as a potential surgical target in oral cavity cancer was explored via topical ex vivo application of the fluorescent tracer EMI-137. Freshly excised tumor specimens obtained from ten patients with squamous cell carcinoma of the tongue were incubated with EMI-137 and imaged with a clinical-grade Cy5 prototype fluorescence camera. In-house developed image processing software allowed video-rate assessment of the tumor-to-background ratio (TBR). Fluorescence imaging results were related to standard pathological evaluation and c-MET immunohistochemistry. After incubation with EMI-137, 9/10 tumors were fluorescently illuminated. Immunohistochemistry revealed c-Met expression in all ten specimens. Non-visualization could be linked to a more deeply situated lesion. Tumor assessment was improved via video representation of the TBR (median TBR: 2.5 (range 1.8–3.1)). Ex vivo evaluation of tumor specimens suggests that c-Met is a possible candidate for fluorescence-guided surgery in oral cavity cancer.

RSPNet: Relative Speed Perception for Unsupervised Video Representation Learning

We study unsupervised video representation learning that seeks to learn both motion and appearance features from unlabeled video only, which can be reused for downstream tasks such as action recognition. This task, however, is extremely challenging due to 1) the highly complex spatial-temporal information in videos and 2) the lack of labeled data for training. Unlike representation learning for static images, it is difficult to construct a suitable self-supervised task to effectively model both motion and appearance features. More recently, several attempts have been made to learn video representation through video playback speed prediction. However, it is non-trivial to obtain precise speed labels for the videos. More critically, the learned models may tend to focus on motion patterns and thus may not learn appearance features well. In this paper, we observe that the relative playback speed is more consistent with motion patterns and thus provides more effective and stable supervision for representation learning. Therefore, we propose a new way to perceive the playback speed and exploit the relative speed between two video clips as labels. In this way, we are able to effectively perceive speed and learn better motion features. Moreover, to ensure the learning of appearance features, we further propose an appearance-focused task, where we enforce the model to perceive the appearance difference between two video clips. We show that jointly optimizing the two tasks consistently improves the performance on two downstream tasks (namely, action recognition and video retrieval) w.r.t the increasing pre-training epochs. Remarkably, for action recognition on the UCF101 dataset, we achieve 93.7% accuracy without the use of labeled data for pre-training, which outperforms the ImageNet supervised pre-trained model. Our code, pre-trained models, and supplementary materials can be found at https://github.com/PeihaoChen/RSPNet.

DramaQA: Character-Centered Video Story Understanding with Hierarchical QA

Despite recent progress on computer vision and natural language processing, developing a machine that can understand video story is still hard to achieve due to the intrinsic difficulty of video story. Moreover, researches on how to evaluate the degree of video understanding based on human cognitive process have not progressed as yet. In this paper, we propose a novel video question answering (Video QA) task, DramaQA, for a comprehensive understanding of the video story. The DramaQA focuses on two perspectives: 1) Hierarchical QAs as an evaluation metric based on the cognitive developmental stages of human intelligence. 2) Character-centered video annotations to model local coherence of the story. Our dataset is built upon the TV drama "Another Miss Oh" and it contains 17,983 QA pairs from 23,928 various length video clips, with each QA pair belonging to one of four difficulty levels. We provide 217,308 annotated images with rich character-centered annotations, including visual bounding boxes, behaviors and emotions of main characters, and coreference resolved scripts. Additionally, we suggest Multi-level Context Matching model which hierarchically understands character-centered representations of video to answer questions. We release our dataset and model publicly for research purposes, and we expect our work to provide a new perspective on video story understanding research.

Semantic Grouping Network for Video Captioning

This paper considers a video caption generating network referred to as Semantic Grouping Network (SGN) that attempts (1) to group video frames with discriminating word phrases of partially decoded caption and then (2) to decode those semantically aligned groups in predicting the next word. As consecutive frames are not likely to provide unique information, prior methods have focused on discarding or merging repetitive information based only on the input video. The SGN learns an algorithm to capture the most discriminating word phrases of the partially decoded caption and a mapping that associates each phrase to the relevant video frames - establishing this mapping allows semantically related frames to be clustered, which reduces redundancy. In contrast to the prior methods, the continuous feedback from decoded words enables the SGN to dynamically update the video representation that adapts to the partially decoded caption. Furthermore, a contrastive attention loss is proposed to facilitate accurate alignment between a word phrase and video frames without manual annotations. The SGN achieves state-of-the-art performances by outperforming runner-up methods by a margin of 2.1%p and 2.4%p in a CIDEr-D score on MSVD and MSR-VTT datasets, respectively. Extensive experiments demonstrate the effectiveness and interpretability of the SGN.

Enhancing Unsupervised Video Representation Learning by Decoupling the Scene and the Motion

One significant factor we expect the video representation learning to capture, especially in contrast with the image representation learning, is the object motion. However, we found that in the current mainstream video datasets, some action categories are highly related with the scene where the action happens, making the model tend to degrade to a solution where only the scene information is encoded. For example, a trained model may predict a video as playing football simply because it sees the field, neglecting that the subject is dancing as a cheerleader on the field. This is against our original intention towards the video representation learning and may bring scene bias on a different dataset that can not be ignored. In order to tackle this problem, we propose to decouple the scene and the motion (DSM) with two simple operations, so that the model attention towards the motion information is better paid. Specifically, we construct a positive clip and a negative clip for each video. Compared to the original video, the positive/negative is motion-untouched/broken but scene-broken/untouched by Spatial Local Disturbance and Temporal Local Disturbance. Our objective is to pull the positive closer while pushing the negative farther to the original clip in the latent space. In this way, the impact of the scene is weakened while the temporal sensitivity of the network is further enhanced. We conduct experiments on two tasks with various backbones and different pre-training datasets, and find that our method surpass the SOTA methods with a remarkable 8.1% and 8.8% improvement towards action recognition task on the UCF101 and HMDB51 datasets respectively using the same backbone.

A unified model for egocentric video summarization: an instance-based approach

Video summarization generates compact representations of videos in the form of summaries. The proposed framework is a unified model for instance-driven egocentric video summarization addressing generic and query-based summarization along with multi-video summarization. The model employs deep learning for object detection and semantic web technologies in the form of ontologies for query inferences. Combining user preferences in the form of object queries has aided in producing summaries that are subjective in nature. Quantitative evaluations performed on two novel datasets namely, ‘vehicle expo’ and ‘academic inspection’ prove that the proposed framework produces remarkable results with the employment of instance-driven modules for summarization. Additional experimental analysis for shot boundary detection have been conducted based on proposed method and conventional methods establishing the significance of the instance-based model. Moreover, qualitative evaluations further ensure that the summaries are concise, representative, diverse and semantically relevant further substantiating the need for instance-driven models in video summarization.

Action Coherence Network for Weakly-Supervised Temporal Action Localization

Weakly-supervised Temporal Action Localization (W-TAL) aims at simultaneously classifying and locating all action instances with only video-level supervision. However, current W-TAL methods have two limitations. First, they ignore the difference in video representations between an action instance and its surrounding background when generating and scoring action proposals. Second, the unique characteristics of the RGB frames and optical flow are largely ignored when fusing these two modalities. To address these problems, an Action Coherence Network (ACN) is proposed in this paper. Its core is a new coherence loss which exploits both classification predictions and video content representations to supervise action boundary regression and thus leads to more accurate action localization results. Besides, the proposed ACN explicitly takes into account the specific characteristics of RGB frames and optical flow by training two separate sub-networks, each of which is able to generate modality-specific action proposals independently. Finally, to take advantage of the complementary action proposals generated by two streams, a novel fusion module is introduced to reconcile them and obtain the final action localization results. Experiments on the THUMOS14 and ActivityNet datasets show that our ACN outperforms the state-of-the-art W-TAL methods, and is even comparable to some recent fully-supervised methods. Particularly, ACN achieves a mean average precision of 26.4% on the THUMOS14 dataset under the IoU threshold 0.5.