Frame-level Features Research Articles

Pseudo-Phoneme Label Loss for Text-Independent Speaker Verification

Compared with text-independent speaker verification (TI-SV) systems, text-dependent speaker verification (TD-SV) counterparts often have better performance for their efficient utilization of speech content information. On this account, some TI-SV methods tried to boost performance by incorporating an extra automatic speech recognition (ASR) component to explore content information, such as c-vector. However, the introduced ASR component requires a large amount of annotated data and consumes high computation resources. In this paper, we propose a pseudo-phoneme label (PPL) loss for the TI-SR task by integrating content cluster loss at the frame level and speaker recognition loss at the segment level in a unified network by multitask learning, without additional data requirement and exhausting computation. By referring to HuBERT, we generate pseudo-phoneme labels to adjust a frame level feature distribution by deep cluster to ensure each cluster corresponds to an implicit pronunciation unit in the feature space. We compare the proposed loss with the softmax loss, center loss, triplet loss, log-likelihood-ratio cost loss, additive margin softmax loss and additive angular margin loss on the VoxCeleb database. Experimental results demonstrate the effectiveness of our proposed method.

Frame-level refinement networks for skeleton-based gait recognition

Gait is considered as a promising biometric feature due to its support for long-distance and non-contact recognition. Most existing skeleton-based gait recognition methods deliver the same topology of skeleton graph for each frame and treat them equally in the process of temporal feature extraction and fusion, which undoubtedly limits the expressive capability of the model. In this paper, we propose a frame-level refinement network to adaptively learn specific topology in different frames and capture long-range dependencies between frames through transformer self-attention. Specifically, we design a frame-level topology refinement graph convolution (FTR-GC) to dynamically model different correlations between joints for each frame. In addition, we introduce transformer self-attention at the frame level, which can learn inter-frame long-range relations between the same joint. Finally, an attention-based frame-level feature aggregation module (FFAM) is presented to produce discriminative global features of input gait sequences. Experiments on the popular public dataset CASIA-B and OUMVLP-Pose show that our method notably surpasses state-of-the-art model-based methods, verifying the effectiveness of the proposed modules.

Online action proposal generation using spatio-temporal attention network

Temporal action proposal generation aims to generate temporal boundaries containing action instances. In real-time applications such as surveillance cameras, autonomous driving, and traffic monitoring, the online localization and recognition of human activities occurring in short temporal intervals are important areas of research. Existing approaches of temporal action proposal generation consider only the offline and frame-level feature aggregation along the temporal dimension. Those offline methods also generate many redundant irrelevant proposal regions in the frames as temporal boundaries. This leads to higher computational cost along with slow processing speed which is not suitable for online tasks. In this study, we propose a novel spatio-temporal attention network for online action proposal generation as opposed to existing offline proposal generation methods. Our novel proposed approach incorporates the inter-dependency between the spatial and temporal context information of each incoming video clip to generate more relevant online temporal action proposals. First, we propose a windowed spatial attention module to capture the inter-spatial relationship between the features of incoming frames. The windowed spatial network produces more robust clip-level feature representation and efficiently deals with noisy features such as occlusion or background scenes. Second, we introduce a temporal attention module to capture relevant temporal dynamic information mutually to the localized spatial information to model the long inter-frame temporal relationship since most online real life videos are untrimmed in nature. By applying these two attention modules sequentially, the novel proposed spatio-temporal network model is able to generate precise action boundaries at a particular instant of time. In addition, the model generates fewer discriminative temporal action proposals while maintaining a low computational cost and high processing speed suitable for online settings.

Exploring Motion and Appearance Information for Temporal Sentence Grounding

This paper addresses temporal sentence grounding. Previous works typically solve this task by learning frame-level video features and align them with the textual information. A major limitation of these works is that they fail to distinguish ambiguous video frames with subtle appearance differences due to frame-level feature extraction. Recently, a few methods adopt Faster R-CNN to extract detailed object features in each frame to differentiate the fine-grained appearance similarities. However, the object-level features extracted by Faster R-CNN suffer from missing motion analysis since the object detection model lacks temporal modeling. To solve this issue, we propose a novel Motion-Appearance Reasoning Network (MARN), which incorporates both motion-aware and appearance-aware object features to better reason object relations for modeling the activity among successive frames. Specifically, we first introduce two individual video encoders to embed the video into corresponding motion-oriented and appearance-aspect object representations. Then, we develop separate motion and appearance branches to learn motion-guided and appearance-guided object relations, respectively. At last, both motion and appearance information from two branches are associated to generate more representative features for final grounding. Extensive experiments on two challenging datasets (Charades-STA and TACoS) show that our proposed MARN significantly outperforms previous state-of-the-art methods by a large margin.

Advanced video anomaly detection using 2D CNN and stacked LSTM with deep active learning-based model

Around the world, the video surveillance system has gained wide acceptance and astonishing growth due to its broad applications. The surveillance system has become a paramount tool and benchmark for analyzing the harmony and safety of society. Anomaly detection and its associated applications play a key role in the integrity of the system. The aim of anomaly detection is to find rare and sparse occurrences of events from videos. Developing an accurate and time-efficient system i s s till remains c hallenging due to t he dynamic nature of a nomalies. The deep learning-based end-to-end system with full use of both spatial and temporal features from the input videos is proposed. The model combines the use of 2DCNN and Stacked LSTM to extract frame-level features through an anisotropic Gunnar Farneback Optical Flow algorithm. The system is evaluated on the benchmarked datasets namely UCSD Ped1 and UCSD Ped2, and it achieves an AUC of 95% and 94% respectively. The experimental results indicate that the proposed method is superior to state-of-the-art algorithms.

Speech intelligibility assessment of dysarthria using Fisher vector encoding

Neuromuscular disorders can lead to dysarthria. Dysarthria is a speech disorder that mainly affects the human motor speech system. It often results in reduced speech intelligibility. Speech intelligibility is one of the parameters to assess the severity of dysarthria. The intelligibility of speech decreases as the severity increases. Automatic speech intelligibility assessment could be reliable and cost-effective compared to the conventional methods which need experienced speech pathologists. Automatic intelligibility assessment includes feature extraction and classification. Features extracted from spectral and cepstral domains are investigated for use in intelligibility assessment. Frame level features are extracted from different signal representations in spectral and cepstral domains. Fisher vector encoding and temporal encoding that uses descriptive statistics are applied to convert frame level features into utterance level. These features are fed to PLDA and ANN classifiers for intelligibility assessment. Overall, the performance of Fisher vector encoding is found to be superior compared to temporal encoding. In all features, the performance of ANN is better in assessing the intelligibility levels compared to PLDA as expected. The STFT, as well as harmonics in spectral domain and CQCC in the cepstral domain, performed well for intelligibility assessment of unseen data of TORGO database.

FRAUG: A FRAME RATE BASED DATA AUGMENTATION METHOD FOR DEPRESSION DETECTION FROM SPEECH SIGNALS.

In this paper, a data augmentation method is proposed for depression detection from speech signals. Samples for data augmentation were created by changing the frame-width and the frame-shift parameters during the feature extraction process. Unlike other data augmentation methods (such as VTLP, pitch perturbation, or speed perturbation), the proposed method does not explicitly change acoustic parameters but rather the time-frequency resolution of frame-level features. The proposed method was evaluated using two different datasets, models, and input acoustic features. For the DAIC-WOZ (English) dataset when using the DepAudioNet model and mel-Spectrograms as input, the proposed method resulted in an improvement of 5.97% (validation) and 25.13% (test) when compared to the baseline. The improvements for the CONVERGE (Mandarin) dataset when using the x-vector embeddings with CNN as the backend and MFCCs as input features were 9.32% (validation) and 12.99% (test). Baseline systems do not incorporate any data augmentation. Further, the proposed method outperformed commonly used data-augmentation methods such as noise augmentation, VTLP, Speed, and Pitch Perturbation. All improvements were statistically significant.

Video Person Re-Identification with Frame Sampling-Random Erasure and Mutual Information-Temporal Weight Aggregation.

Partial occlusion and background clutter in camera video surveillance affect the accuracy of video-based person re-identification (re-ID). To address these problems, we propose a person re-ID method based on random erasure of frame sampling and temporal weight aggregation of mutual information of partial and global features. First, for the case in which the target person is interfered or partially occluded, the frame sampling–random erasure (FSE) method is used for data enhancement to effectively alleviate the occlusion problem, improve the generalization ability of the model, and match persons more accurately. Second, to further improve the re-ID accuracy of video-based persons and learn more discriminative feature representations, we use a ResNet-50 network to extract global and partial features and fuse these features to obtain frame-level features. In the time dimension, based on a mutual information–temporal weight aggregation (MI–TWA) module, the partial features are added according to different weights and the global features are added according to equal weights and connected to output sequence features. The proposed method is extensively experimented on three public video datasets, MARS, DukeMTMC-VideoReID, and PRID-2011; the mean average precision (mAP) values are 82.4%, 94.1%, and 95.3% and Rank-1 values are 86.4%, 94.8%, and 95.2%, respectively.

Vision Transformer and Deep Sequence Learning for Human Activity Recognition in Surveillance Videos

Human Activity Recognition is an active research area with several Convolutional Neural Network (CNN) based features extraction and classification methods employed for surveillance and other applications. However, accurate identification of HAR from a sequence of frames is a challenging task due to cluttered background, different viewpoints, low resolution, and partial occlusion. Current CNN-based techniques use large-scale computational classifiers along with convolutional operators having local receptive fields, limiting their performance to capture long-range temporal information. Therefore, in this work, we introduce a convolution-free approach for accurate HAR, which overcomes the above-mentioned problems and accurately encodes relative spatial information. In the proposed framework, the frame-level features are extracted via pretrained Vision Transformer; next, these features are passed to multilayer long short-term memory to capture the long-range dependencies of the actions in the surveillance videos. To validate the performance of the proposed framework, we carried out extensive experiments on UCF50 and HMDB51 benchmark HAR datasets and improved accuracy by 0.944% and 1.414%, respectively, when compared to state-of-the-art deep models.

Multi-label video classification via coupling attentional multiple instance learning with label relation graph

Multi-label video classification is a challenging problem in pattern recognition field, as it is difficult to grasp the occurring localizations of a huge number of labels in videos. To solve this problem, we propose a general framework named MALL-CNN, i.e., Multi-Attention Label Relation Learning Convolutional Neural Network. MALL-CNN not only builds the correspondences between labels and videos by an attention mechanism, but also captures label co-occurrence by a graph learning approach. Specifically, we introduce multiple instance learning to composite a set of frame-level features into a video-level feature. Then, video-level feature is mapped into the content-aware category representations in an improved attentional manner. Further, these representations are enhanced by a series of label relation graphs, which transform global label relationships to the label relationships of current video. With the three processes, frame feature aggregation, video feature mapping, and label relationship construction can be achieved in MALL-CNN for multi-label video classification. Extensive experiments on real-world scene benchmark Youtube-8M verify that MALL-CNN with only frame feature surpasses the state of the arts with multi-modal features as well as ensemble models.

Class-GE2E: Speaker Verification Using Self-Attention and Transfer Learning with Loss Combination

Recent studies prove that speaker verification performance improves by employing an attention mechanism compared to using temporal and statistical pooling techniques. This paper proposes an advanced multi-head attention method, which utilizes a sorted vector of the frame-level features to consider a higher correlation. In this study, we also propose a transfer learning scheme to maximize the effectiveness of the two loss functions, which are the classifier-based cross entropy loss function and metric-based GE2E loss function, to learn the distance between embeddings. The sorted multi-head attention (SMHA) method outperforms the conventional attention methods showing 4.55% in equal error rate (EER). The proposed transfer learning scheme with Class-GE2E loss function significantly improved our attention-based systems. In particular, the EER of the SMHA decreased to 4.39% by employing transfer learning with Class-GE2E loss. The experimental results demonstrate that our effort to include a greater correlation between frame-level features for multi-head attention processing, and the combining of two different loss functions through transfer learning, is highly effective for improving speaker verification performance.

Relation-based global-partial feature learning network for video-based person re-identification

Video-based person re-identification (Re-ID) aims to match the same pedestrian from the video sequences captured by non-overlapping cameras. It is the key to fully extracting abundant spatial and temporal information from the video frames in video-based Re-ID. In this paper, a novel Relation-Based Global-Partial Feature Learning framework is proposed to explore discriminative spatiotemporal features with the help of the global and partial relationship between frames. Specifically, we propose a Relation-Based Global Feature Learning Module (RGL) to obtain global references for generating features correlation maps between frames in the video sequence and determine the importance of frame-level features. As the supplementary of the global relation-based features, a Relation-Based Partial Feature Learning Module (RPL) is also proposed to obtain the relationship between partial features of the same spatial position in different frames to enhance the frame-level partial representation. Moreover, we design a multi-level training scheme to deeply supervise our model. Extensive experiments are conducted on three public video-based person Re-ID datasets, and the results indicate that our framework achieves state-of-the-art performance on three benchmarks.

Two-stage streaming keyword detection and localization with multi-scale depthwise temporal convolution

A keyword spotting (KWS) system running on smart devices should accurately detect the appearances and predict the locations of predefined keywords from audio streams, with small footprint and high efficiency. To this end, this paper proposes a new two-stage KWS method which combines a novel multi-scale depthwise temporal convolution (MDTC) feature extractor and a two-stage keyword detection and localization module. The MDTC feature extractor learns multi-scale feature representation efficiently with dilated depthwise temporal convolution, modeling both the temporal context and the speech rate variation. We use a region proposal network (RPN) as the first-stage KWS. At each frame, we design multiple time regions, which all take the current frame as the end position but have different start positions. These time regions (or formally anchors) are used to indicate rough location candidates of keyword. With frame level features from the MDTC feature extractor as inputs, RPN learns to propose keyword region proposals based on the designed anchors. To alleviate the keyword/non-keyword class imbalance problem, we specifically introduce a hard example mining algorithm to select effective negative anchors in RPN training. The keyword region proposals from the first-stage RPN contain keyword location information which is subsequently used to explicitly extract keyword related sequential features to train the second-stage KWS. The second-stage system learns to classify and transform region proposal to keyword IDs and ground-truth keyword region respectively. Experiments on the Google Speech Command dataset show that the proposed MDTC feature extractor surpasses several competitive feature extractors with a new state-of-the-art command classification error rate of 1.74%. With the MDTC feature extractor, we further conduct wake-up word (WuW) detection and localization experiments on a commercial WuW dataset. Compared to a strong baseline, our proposed two-stage method achieves relatively 27–32% better false rejection rate at one false alarm per hour, while for keyword localization, the two-stage approach achieves more than 0.95 mean intersection-over-union ratio, which is clearly better than the one-stage RPN method.

Attention-Based Temporal-Frequency Aggregation for Speaker Verification.

Convolutional neural networks (CNNs) have significantly promoted the development of speaker verification (SV) systems because of their powerful deep feature learning capability. In CNN-based SV systems, utterance-level aggregation is an important component, and it compresses the frame-level features generated by the CNN frontend into an utterance-level representation. However, most of the existing aggregation methods aggregate the extracted features across time and cannot capture the speaker-dependent information contained in the frequency domain. To handle this problem, this paper proposes a novel attention-based frequency aggregation method, which focuses on the key frequency bands that provide more information for utterance-level representation. Meanwhile, two more effective temporal-frequency aggregation methods are proposed in combination with the existing temporal aggregation methods. The two proposed methods can capture the speaker-dependent information contained in both the time domain and frequency domain of frame-level features, thus improving the discriminability of speaker embedding. Besides, a powerful CNN-based SV system is developed and evaluated on the TIMIT and Voxceleb datasets. The experimental results indicate that the CNN-based SV system using the temporal-frequency aggregation method achieves a superior equal error rate of 5.96% on Voxceleb compared with the state-of-the-art baseline models.

Video Domain Adaptation based on Optimal Transport in Grassmann Manifolds

Domain adaptation is a fundamental research field, which focuses on transforming knowledge between different domains. With the massive growth of video data, the video domain adaptation problem becomes increasingly significant for practical tasks. Motivated by the excellent performance of Grassmann manifolds representation in video recognition tasks, we propose an optimal transport based video domain adaptation model on Grassmann manifolds. The proposed model reduces the discrepancy between different domains for the frame and video level features. First, the frame level discrepancy is reduced by extracting domain consistency features. At the video level, a fixed number of frame features are formed and represented as points on Grassmann manifolds. These points are fused with predicted labels to form fusion features. Finally, the video level discrepancy is reduced by minimizing the distribution discrepancy of the fusion features between two domains. Cross-domain video recognition experiments demonstrate the validity of the proposed model. The experimental results demonstrate the excellent performance of the proposed algorithm compared with the state-of-art video domain adaptation models.

Explicit kissing scene detection in cartoon using convolutional long short-term memory

The main concern of this study is due to certain cartoon content consisting of explicit scenes such as kissing, sex, violence. That are somehow not suitable for kids and may contradict to some religions and cultures. There are some reasons the film industry does not expel the kissing scene in a cartoon movie. It is categorized as a romance sequence and love scene. These could be a double-edged weapon that will ruin an individual’s childhood through excessive exposure to explicit content. This paper proposes a deep learning-based classifier to detect the kissing scene in the cartoon by using Darknet-19 for frame-level feature extraction, while the feature aggregation in the temporal domain is using convolutional long short-term memory (conv-LSTM). This paper also has discussed a few steps related to evaluation and analysis regarding the performance of the models. Extensive experiments prove that the proposed system provides excellent results of 96.43% accuracy to detect the kissing scene in the cartoon. Due to high accuracy performance, the model is suitable to be a kissing scene filter feature in a digital video player that may able to decrease the excessive exposure to explicit content for kids.

FeatInter: Exploring fine-grained object features for video-text retrieval

In this paper, we target the challenging task of video-text retrieval. The common way for this task is to learn a text-video joint embedding space by cross-modal representation learning, and compute the cross-modality similarity in the joint space. As videos typically contain rich information, how to represent videos in a joint embedding space is crucial for video-text retrieval. The majority of works typically depend on pre-extracted frame-level features or clip-level features for video representation, which may cause fine-grained object information in videos to be ignored. To alleviate it, we explicitly introduce more fine-grained object-level features to enrich video representation. In order to exploit the potential of the object-level features, we propose a new model named FeatInter, which jointly considers the visual and semantic features of objects. Besides, a visual-semantic interaction and a cross-feature interaction are proposed to mutually enhance object features and frame features. Extensive experiments on two challenging video datasets, i.e., MSR-VTT and TGIF, demonstrate the effectiveness of our proposed model. Moreover, our model achieves a new state-of-the-art on TGIF. While the state-of-the-art methods use seven video features on MSR-VTT, our model with just three features obtains comparable performance.

Mixture Representation Learning for Deep Speaker Embedding

How to effectively convert a sequence of variable-length acoustic features to a fixed-dimension representation has always been a research focus in speaker recognition. In state-of-the-art speaker recognition systems, the conversion is implemented by concatenating the mean and the standard deviation of a sequence of frame-level features. However, a single mean and a single standard deviation are limited descriptive statistics for an acoustic sequence even with powerful feature extractors such as convolutional neural networks. In this paper, we propose a novel statistics pooling method that can produce more descriptive statistics through a mixture representation. Our approach is inspired by the expectation–maximization (EM) algorithm in Gaussian mixture models (GMMs). Instead of using traditional GMM style alignment, we novelly leverage modern deep learning tools to produce a more powerful mixture representation. The novelty includes: (1) unlike GMMs, the mixture assignments are determined by an attention network instead of the Euclidean distances between the frame-level features and explicit centers; (2) instead of using a single frame as input to the attention network, contextual frames are included to smooth out attention transition; and (3) soft-attention assignments are replaced by hard-attention assignments via the Gumbel-Softmax with straight-through estimators. With the proposed attention mechanism, we obtained a 13.7% relative improvement over vanilla mean and standard deviation pooling in the VOiCES19-eval set.

Spatial-Temporal Graphs for Cross-Modal Text2Video Retrieval

Cross-modal text to video retrieval aims to find relevant videos for given text queries, which is crucial for various real-world applications. The key to address this task is to build the correspondence between video and text such that the related samples from different modalities can be aligned. As the text (sentence) contains both nouns and verbs representing objects as well as their interactions, retrieving the relevant videos requires a fine-grained understanding of video contents---not only the semantic concepts (i.e., objects) but also the interactions between them. Nevertheless, the current approaches mostly represent videos with aggregated frame-level features for the learning of joint space and ignore the information of object interactions, which usually results in suboptimal retrieval performance. To improve the performance of cross-modal video retrieval, this paper proposes a framework that models videos as spatial-temporal graphs where nodes correspond to visual objects and edges correspond to the relations/interactions between objects. With the spatial-temporal graphs, the object interactions in frame sequences can be captured to enrich the video representations for joint space learning. Specifically, Graph Convolutional Network is introduced to learn the representations on spatial-temporal graphs, aiming to encode the spatial-temporal interactions between objects; while BERT is introduced to dynamically encode the sentence according to the context for cross-modal retrieval. Extensive experiments verify the effectiveness of the proposed framework and it achieves promising performances on both MSR-VTT and LSMDC datasets.

DeltaVLAD: An efficient optimization algorithm to discriminate speaker embedding for text-independent speaker verification

<abstract> <p>Text-independent speaker verification aims to determine whether two given utterances in open-set task originate from the same speaker or not. In this paper, some ways are explored to enhance the discrimination of embeddings in speaker verification. Firstly, difference is used in the coding layer to process speaker features to form the DeltaVLAD layer. The frame-level speaker representation is extracted by the deep neural network with differential operations to calculate the dynamic changes between frames, which is more conducive to capturing insignificant changes in the voiceprint. Meanwhile, NeXtVLAD is adopted to split the frame-level features into multiple word spaces before aggregating, and subsequently perform VLAD operations in each subspace, which can significantly reduce the number of parameters and improve performance. Secondly, the margin-based softmax loss function and the few-shot learning-based loss function are proposed to be combined for more discriminative speaker embeddings. Finally, for a fair comparison, the experimental results are performed on Voxceleb-1 showing superior performance of speaker verification system and can obtain new state-of-the-art results.</p> </abstract>