Time-sync Comments Research Articles

Fragment of Interest: Personalized Video Fragment Recommendation with Inter-Fragment & Intra-Fragment Contextual Effect

In today’s fast-paced digital landscape, the attention span of users consuming video content is alarmingly brief, often as short as 15 seconds for music or entertainment videos and 6 minutes for lecture videos. This presents a significant challenge for video producers and platform providers as they seek to engage users with longer content. One promising solution involves recommending specific fragments within longer videos that align with individual user profiles. In this paper, we address this challenge by introducing a novel framework for video fragment recommendations, guided by three key insights. First, we implement a Self-Attention Block that captures the inter-fragment contextual effect, enhancing the relevance of recommendations. Second, we incorporate video-level preferences to ensure that the fragment recommendations are consistent with users’ overall interests. Third, we propose a Self-Attentive Herding Effect (SAHE) module to model the intra-fragment contextual effect, specifically the herding effect of time-sync comments within a fragment. To evaluate the effectiveness of our proposed method, we conduct extensive experiments comparing our model against the state-of-the-art approaches in terms of NDCG@K and Recall@K. Our results demonstrate that the model effectively leverages inter-fragment, intra-fragment contextual effects, and video-level preferences, outperforming existing methods. Additionally, we carry out empirical experiments to analyze the key components and parameters of the proposed model, providing further insights into its performance.

Sentiment classification of time-sync comments: A semi-supervised hierarchical deep learning method

Time-sync comment (TSC) has emerged as a new type of textual comment for real-time user interactions on online video platforms. The sentiment classification of TSCs provides considerable potential for platforms to optimize operation strategies but inevitably faces great challenges due to the TSCs’ often uninformative and informal text. Considering the contextual dependency among TSCs posted within the same video clip, this study posits that contextual TSCs may benefit the sentiment classification of a target TSC. To address the challenges of leveraging contextual TSCs, such as their semantic representation and fusion, we propose a semi-supervised hierarchical deep learning method for the sentiment classification of TSCs. We design a hierarchical architecture to capture the semantics of TSCs at the word, comment, and context levels. Considering the varying importance of words and comments, we also design attention mechanisms to focus on important sentiment information and fuse semantic representations. Empirical evaluation shows that the proposed method outperforms benchmarked sentiment classification methods. This study advances our knowledge of contextual information indicative of TSC sentiment, and contributes to improving the service operation of online video platforms.

Comprehending the Gossips: Meme Explanation in Time-Sync Video Comment via Multimodal Cues

Recent years have witnessed the booming of online social media platforms with embracing the popular service called “Time-Sync Comment”, which supports the viewers to share their time-sync opinions along with video content. In this way, we observe that numerous semantically-altered terms, or “Memes”, were created by niche users to express their unique ideas and emotions, and further attracted a large group of viewers with better activity and enthusiasm. Unfortunately, since the memes were created based on domain-specific knowledge and semantically varied depending on the multimodal context in videos, newcomers may fail to comprehend the semantic connotation of memes, which may severely impair their user-experiences. To deal with this issue, in this article, we propose a novel meme explanation framework, called ProMDE, to automatically capture and comprehend the memes in time-sync comments, which could further benefit the viewers with meme explanation service. Specifically, we first iteratively reconstruct the original time-sync comments compared with visual embedding to detect the semantically-altered terms as meme candidates. Afterward, based on the guides from the domain-specific corpus, visual and textual features will be fused to represent the context-aware multimodal cues. Moreover, to accurately describe the commonly-seen homophones in memes, i.e., they have the same pronunciation but different word-spelling expressions, we integrate the phonetic symbols as an additional modality to enhance the framework. Finally, we utilize a Transformer-based decoder to generate the natural language explanation for captured memes. Extensive experiments on a large real-world dataset prove that our framework could significantly outperform several state-of-the-art baseline methods, demonstrating the efficacy of modeling multimodal context and pronunciation for meme detection and explanation.

Sentiment Analysis on Online Videos by Time-Sync Comments.

Video highlights are welcomed by audiences, and are composed of interesting or meaningful shots, such as funny shots. However, video shots of highlights are currently edited manually by video editors, which is inconvenient and consumes an enormous amount of time. A way to help video editors locate video highlights more efficiently is essential. Since interesting or meaningful highlights in videos usually imply strong sentiments, a sentiment analysis model is proposed to automatically recognize sentiments of video highlights by time-sync comments. As the comments are synchronized with video playback time, the model detects sentiment information in time series of user comments. Moreover, in the model, a sentimental intensity calculation method is designed to compute sentiments of shots quantitatively. The experiments show that our approach improves the F1 score by 12.8% and overlapped number by 8.0% compared with the best existing method in extracting sentiments of highlights and obtaining sentimental intensities, which provides assistance for video editors in editing video highlights efficiently.

CoEvo-Net: Coevolution Network for Video Highlight Detection

Video highlight detection (VHD) has emerged as a pressing task due to the unprecedentedly increasing amount of video data, such as those from e-commerce live-broadcasting platforms. Many approaches focus on exploiting text data, in the form of video description or time-sync comments, to facilitate the VHD task. Despite the promising results, they have largely overlooked the noises inherent in the text data and have mostly relied on isolating the feature of text and video. In this paper, we introduce a novel model to handle VHD, termed Coevolution Network (CoEvo-Net), that allows us to account for joint learning of the language and video features explicitly via a coevolution paradigm, in which features from the two data modalities progressively refine each other. This is achieved by a dedicated CoEvo-Cell that takes language and video together as inputs, extracts cross-modality, and filters the undesired parts of the input, such as words in a sentence. Furthermore, we release a large-scale dataset of e-commerce for VHD, in which each video is coupled with a sentence for description, to benchmark the sentence-based VHD approaches. Extensive experiments on the released dataset demonstrate that CoEvo-Net achieves state-of-the-art performance. Our dataset and code will be made publicly available.

Encrypted Live Streaming Channel Identification With Time-Sync Comments

The time-sync comments have been prevalent in modern live streaming systems to provide a real-time interaction experience for viewers. Whereas, the time-sync comments traffic can also act as a delicate fingerprint of encrypted live channels, leading to potential risks of privacy leakage. Most of previous video channel identification strategies with video bitrate-based fingerprint presume strict requirements on the implementation environments, which often assume that there is no interference from irrelevant traffic flows or network conditions. However, the time-sync comments sessions are distinct from other irrelevant traffic flows, and the traffic pattern is resilient to various network conditions, e.g., bandwidth limitation and transmission delay. In this paper, we design a system for encrypted live channel identification with time-sync comments traffic analysis. Specifically, both the inter-application and inner-application traffic filters are proposed to eliminate the irrelevant traffic flows, respectively. Further, a comment rate estimation method is developed through investigation of relationship between comment number, comment length and packet length. Finally, the dynamic time warping(DTW) algorithm is improved for similarity matching in delay tolerant environment. In order to evaluate the system performance, we setup the prototype system with AWS EC2 server and utilize the real world trace data from Youtube and BiliBili. The experimental results show that the accuracy of the filter can reach 93.2%, and the accuracy of the comment rate estimation method can reach up to 91%. The match accuracy between fingerprint and comment rate can reach 92.1% within 200 seconds eavesdropping, which is 2% higher than using bitrate fingerprint and traffic pattern in the latest research, and can be increased to 98.2% when the eavesdropping time extends to 500 seconds.

GIF Thumbnails: Attract More Clicks to Your Videos

With the rapid increase of mobile devices and online media, more and more people prefer posting/viewing videos online. Generally, these videos are presented on video streaming sites with image thumbnails and text titles. While facing huge amounts of videos, a viewer clicks through a certain video with high probability because of its eye-catching thumbnail. However, current video thumbnails are created manually, which is time-consuming and quality-unguaranteed. And static image thumbnails contain very limited information of the corresponding videos, which prevents users from successfully clicking what they really want to view. In this paper, we address a novel problem, namely GIF thumbnail generation, which aims to automatically generate GIF thumbnails for videos and consequently boost their Click-Through-Rate (CTR). Here, a GIF thumbnail is an animated GIF file consisting of multiple segments from the video, containing more information of the target video than a static image thumbnail. To support this study, we build the first GIF thumbnails benchmark dataset that consists of 1070 videos covering a total duration of 69.1 hours, and 5394 corresponding manually-annotated GIFs. To solve this problem, we propose a learning-based automatic GIF thumbnail generation model, which is called Generative Variational Dual-Encoder (GEVADEN). As not relying on any user interaction information (e.g. time-sync comments and real-time view counts), this model is applicable to newly-uploaded/rarely-viewed videos. Experiments on our built dataset show that GEVADEN significantly outperforms several baselines, including video-summarization and highlight-detection based ones. Furthermore, we develop a pilot application of the proposed model on an online video platform with 9814 videos covering 1231 hours, which shows that our model achieves a 37.5% CTR improvement over traditional image thumbnails. This further validates the effectiveness of the proposed model and the promising application prospect of GIF thumbnails.

Time-sync comments denoising via graph convolutional and contextual encoding

Time-Sync Comments (TSC), which is a new kind of textual comments on online video websites, has showed its great potential in fine-grain video analysis. However, as a crowd-sourced resource, there are many low quality comments in TSC data and this is an impediment to make full use of TSC. Thus a denoising method is necessary when we are dealing with these comments. In this study, we propose GCCED, a graph convolutional and contextual encoding denoising model for TSC semantic denoising problem. A TSC graph is built on the whole corpus and semantic embedding of words are learned through graph convolution. Moreover, we exploit the relations between TSC and its context and design an embedding method based on the word graph. Experiments on real world dataset are conducted and the result demonstrate the proposed model outperforming other baselines in almost all classification metrics.

Discovering attractive segments in the user-generated video streams

With the rapid development of digital equipment and the continuous upgrading of online media, a growing number of people are willing to post videos on the web to share their daily lives (Jelodar, Paulius, & Sun, 2019). Generally, not all video segments are popular with audiences, some of which may be boring. If we can predict which segment in a newly generated video stream would be popular, the audiences can only enjoy this segment rather than watch the whole video to find the funny point. And if we can predict the emotions that the audiences would induce when they watch a video, this must be helpful for video analysis and for guiding the video-makers to improve their videos. In recent years, crowd-sourced time-sync video comments have emerged worldwide, supporting further research on temporal video labeling. In this paper, we propose a novel framework to achieve the following goal: Predicting which segment in a newly generated video stream (hasn’t been commented with the time-sync comments) will be popular among the audiences. At last, experimental results on real-world data demonstrate the effectiveness of the proposed framework and justify the idea of predicting the popularities of segments in a video exploiting crowd-sourced time-sync comments as a bridge to analyze videos.

Time-Sync Video Tag Extraction Using Semantic Association Graph

Time-sync comments (TSCs) reveal a new way of extracting the online video tags. However, such TSCs have lots of noises due to users’ diverse comments, introducing great challenges for accurate and fast video tag extractions. In this article, we propose an unsupervised video tag extraction algorithm named Semantic Weight-Inverse Document Frequency (SW-IDF). Specifically, we first generate corresponding semantic association graph (SAG) using semantic similarities and timestamps of the TSCs. Second, we propose two graph cluster algorithms, i.e., dialogue-based algorithm and topic center-based algorithm, to deal with the videos with different density of comments. Third, we design a graph iteration algorithm to assign the weight to each comment based on the degrees of the clustered subgraphs, which can differentiate the meaningful comments from the noises. Finally, we gain the weight of each word by combining Semantic Weight (SW) and Inverse Document Frequency (IDF). In this way, the video tags are extracted automatically in an unsupervised way. Extensive experiments have shown that SW-IDF (dialogue-based algorithm) achieves 0.4210 F1-score and 0.4932 MAP (Mean Average Precision) in high-density comments, 0.4267 F1-score and 0.3623 MAP in low-density comments; while SW-IDF (topic center-based algorithm) achieves 0.4444 F1-score and 0.5122 MAP in high-density comments, 0.4207 F1-score and 0.3522 MAP in low-density comments. It has a better performance than the state-of-the-art unsupervised algorithms in both F1-score and MAP.

Video clip recommendation model by sentiment analysis of time-sync comments

With the advent of video time-sync comments, users can not only comment the videos on the Internet, but also share their feelings with others. However, the number of the videos on the Internet is so huge that users do not have enough time and energy to watch all the videos. How to recommend the videos suitable for users has become an important problem. The traditional video sentiment analysis methods can not work effectively and the results are not easy to explain. In this paper, an emotion recognition algorithm based on sync-time comments is proposed, as a basis for the recommendation of video clips. First, we propose a formal description of video clips recommendation based on sentiment analysis. Secondly, by constructing the classification of time-sync comments based on Latent Dirichlet Allocation (LDA) topic model, we evaluate the emotion vector of the words in time-sync comments. Meanwhile, the video clips are recommended according to the emotion relationships among the video clips. The experimental results show that the proposed model is effective in analyzing the complex sentiment of different kinds of text information.

Stories That Big Danmaku Data Can Tell as a New Media

In online video communities, an emerging source of comment, such as danmaku, allows viewers to interact when watching. Prior work discussed the feasibility of application using danmaku, while the comprehensive analysis of large-scale data is vacant to be filled in. We here release our danmaku data collection and report interesting observed phenomena in the danmaku comments. This analysis reveals the temporal distributions and user’s access patterns of online time-sync comments. In particular, we distribute two novel natural language processing (NLP) tasks based on our danmaku dataset and provide baseline models. In the first task, we show how the naive models predict positive or negative sentiment given a danmaku comment, which effectively extends the real applications such as opinion poll prediction and marketing investigation. In the second task, we propose to use the NLP summarization model to make video tagging and summarization. The experimental results suggest that danmaku can not only support deeper and richer interactions between viewers and videos but also with high research value.

Bridging Video Content and Comments: Synchronized Video Description with Temporal Summarization of Crowdsourced Time-Sync Comments

With the rapid growth of online sharing media, we are facing a huge collection of videos. In the meantime, due to the volume and complexity of video data, it can be tedious and time consuming to index or annotate videos. In this paper, we propose to generate temporal descriptions of videos by exploiting the information of crowdsourced time-sync comments which are receiving increasing popularity on many video sharing websites. In this framework, representative and interesting comments of a video are selected and highlighted along the timeline, which provide an informative description of the video in a time-sync manner. The challenge of the proposed application comes from the extremely informal and noisy nature of the comments, which are usually short sentences and on very different topics. To resolve these issues, we propose a novel temporal summarization model based on the data reconstruction principle, where representative comments are selected in order to best reconstruct the original corpus at the text level as well as the topic level while incorporating the temporal correlations of the comments. Experimental results on real-world data demonstrate the effectiveness of the proposed framework and justify the idea of exploiting crowdsourced time-sync comments as a bridge to describe videos.

Reading the Videos: Temporal Labeling for Crowdsourced Time-Sync Videos Based on Semantic Embedding

Recent years have witnessed the boom of online sharing media contents, which raise significant challenges in effective management and retrieval. Though a large amount of efforts have been made, precise retrieval on video shots with certain topics has been largely ignored. At the same time, due to the popularity of novel time-sync comments, or so-called "bullet-screen comments", video semantics could be now combined with timestamps to support further research on temporal video labeling. In this paper, we propose a novel video understanding framework to assign temporal labels on highlighted video shots. To be specific, due to the informal expression of bullet-screen comments, we first propose a temporal deep structured semantic model (T-DSSM) to represent comments into semantic vectors by taking advantage of their temporal correlation. Then, video highlights are recognized and labeled via semantic vectors in a supervised way. Extensive experiments on a real-world dataset prove that our framework could effectively label video highlights with a significant margin compared with baselines, which clearly validates the potential of our framework on video understanding, as well as bullet-screen comments interpretation.