Movie Summarization Research Articles

Human attention based movie summarization: Dataset and baseline model

A movie summarization model can automatically edit a condensed version of a movie by selecting keyframes. Some previous works have proposed some movie summarizers based on traditional methods or recent neural networks and achieved some progress. Despite the demonstrated successes, there are some limitations: (1) previous works mainly resort to hand-crafted heuristics and most of them are unsupervised; (2) currently there is no publicly suitable dataset available for the supervised movie summarization; (3) existing works only focus on the movies themselves while neglecting the audiences, who have the most to say in which part of the movie is more attractive. To break through the aforementioned limitations, we establish a movie summarization dataset Movie50 and propose a novel human attention based annotation pipeline. Furthermore, we propose the A/V-MSNet, an audiovisual neural network that takes advantage of spatio-temporal visual and auditory information to better simulate human attention as well as exploit more plentiful information. The network is designed, trained end-to-end, and evaluated on the public dataset and our dataset. Extensive experiments demonstrate the superiority of the proposed method.

QuickLook: Movie summarization using scene-based leading characters with psychological cues fusion

Due to recent advances in the film industry, the production of movies has grown exponentially, which has led to challenges in what is referred to as discoverability: given the overwhelming number of choices, choosing which film to watch has become a tedious task for audiences. Movie summarization (MS) could help, as it presents the central theme of the movie in a compact format and makes browsing more efficient for the audience. In this paper, we present an automatic MS framework coined as ‘QuickLook’, which identifies the leading characters and fuses multiple cues extracted from a movie. Firstly, the movie data is preprocessed for its division into scenes, followed by shot segmentation. Secondly, the leading characters in each segmented scene are determined. Next, four visual cues that capture the film's scenic beauty, memorability, informativeness and emotional resonance are extracted from shots containing the leading characters. These extracted features are then intelligently fused based on the assignment of different weights; shots with a fusion score above a certain threshold are selected for the final summary. The proposed MS framework is assessed by comparison with official trailers from ten Hollywood movies, providing a novel baseline for future fair comparison in the MS literature. The proposed framework is shown to outperform other state-of-the-art MS methods in terms of enjoyability and informativeness.

Cooking Movie Summarization Based on Recipe Images and Descriptions

Cooking Movie Summarization Based on Recipe Images and Descriptions

Performance Analysis of Video On-demand and Live Video Streaming using Cloud based Services

The advent of Cyber-Physical Systems (CPS) has brought a revolutionary change coined as a mixture of information, communication, computation, and control. With applications in smart grid, health monitoring, automatic avionics, distributed robotics, etc., CPS is currently an area of attention among the academia and industry. The advancement of mobile communications and embedded technology has made it possible to build large scale CPS consisting of the interconnection of mobile phones. These devices collect information about the surrounding environment at any time anywhere basis through real-time video capture. Video streaming has proven to be a massive industry that is growing rapidly playing an important role in everyday life. Customer-driven approach wanting best experience with quality has to be the core offering of contemporary scenario. Video streaming is categorized into Video-On-Demand Streaming (VoDS) and Live Video Streaming (LVS) showing the current state-of-art opportunities. Many diverse applications of video streaming are military video surveillance using drones, live sports match player face recognition, on-demand video characters recognition, movie summarization like identifying parts of the movie which are viewed many times by different users, movie and series recognition, motion detection, gesture recognition, image segmentation, etc. This paper introduces an approach to develop video analysis on VoD and LVS using cloud-based services and analyzes the impact of Quality of Experience (QoE), cost, and bandwidth on the cloud. To achieve the best user experience for video streaming and video analysis, Content Delivery Network (CDN) offers the best QoE at various analyzed locations using various cloud providers like Amazon Web Services (AWS) CloudFront, Google Cloud CDN, Azure CDN, Akamai CDN, etc.

An Improved Audio Based Key Frame Detection Model for Movie Summarization

In the recent scenario, the amount of multimedia content mainly videos is increasing at a high rate. This makes the browsing, retrieval and delivery of videos a very slow and difficult task. Video summarization helps in faster browsing and content indexing of large videos by producing an abstract overview of the whole video. Movie summarization condenses a full length movie into a summary which includes the relevant contents in the movie. The proposed framework implements a movie summarization method based on the audio and video features of a movie. The method comprises of three phases. In the pre-processing phase, audio and frames of the video are extracted and stored. Next in the audio feature analysis phase, audio data having high frequency value are extracted. Three techniques are proposed for audio frame analyses which are high frequency method (HFM), Low frequency method (LFM) and threshold frequency method (TFM). In summary generation phase, key frames are extracted which is further refined in the post processing phase. The experimental results indicate that in majority of the cases HFM achieves better performance than LFM and TFM in both qualitative and quantitative analysis.

Datasets column

Affective video content analysis aims at the automatic recognition of emotions elicited by videos. It has a large number of applications, including mood based personalized content recommendation, video indexing, and efficient movie visualization and browsing. Beyond the analysis of existing video material, affective computing techniques can also be used to generate new content, e.g., movie summarization, personalized soundtrack recommendation to make user-generated videos more attractive. Affective techniques can furthermore be used to enhance the user engagement with advertising content by optimizing the way ads are inserted inside videos.

Personalized Movie Summarization Using Deep CNN‐Assisted Facial Expression Recognition

Personalized movie summarization is demand of the current era due to an exponential growth in movies production. The employed methods for movies summarization fail to satisfy the user’s requirements due to the subjective nature of movies data. Therefore, in this paper, we present a user‐preference based movie summarization scheme. First, we segmented movie into shots using a novel entropy‐based shots segmentation mechanism. Next, temporal saliency of shots is computed, resulting in highly salient shots in which character faces are detected. The resultant shots are then forward propagated to our trained deep CNN model for facial expression recognition (FER) to analyze the emotional state of the characters. The final summary is generated based on user‐preferred emotional moments from the seven emotions, i.e., afraid, angry, disgust, happy, neutral, sad, and surprise. The subjective evaluation over five Hollywood movies proves the effectiveness of our proposed scheme in terms of user satisfaction. Furthermore, the objective evaluation verifies the superiority of the proposed scheme over state‐of‐the‐art movie summarization methods.

Movie indexing and summarization using social network techniques

Movie summarization and indexing is the study which takes into account the understanding of the audiences. Besides, movie summarization focuses on reducing the length of a movie. Regarding this work, we propose a character network analysis to index and summarize the given movie. The method is based on the discovery and analysis of characters with respect to their appearance and the relationships among them in the movie. The strategy analysis is used to detect scenes, to segment the sub-plots of the story, and to extract character network and the main storyline of the movie. As a result, the social strength of each character in the social network is measured using measurement techniques that will then be used to mine the main plotline of the movie. In the final stage, the main storyline is used to provide a summarized version of the movie based on the social power of the characters. Experiments were carried out with 17 series of the Star Wars, the Lord of the Rings and the Harry Potter. The experimental evaluation results show that this study should index and extract summarization versions while keeping the understanding of the audiences.

COGNIMUSE: a multimodal video database annotated with saliency, events, semantics and emotion with application to summarization

Research related to computational modeling for machine-based understanding requires ground truth data for training, content analysis, and evaluation. In this paper, we present a multimodal video database, namely COGNIMUSE, annotated with sensory and semantic saliency, events, cross-media semantics, and emotion. The purpose of this database is manifold; it can be used for training and evaluation of event detection and summarization algorithms, for classification and recognition of audio-visual and cross-media events, as well as for emotion tracking. In order to enable comparisons with other computational models, we propose state-of-the-art algorithms, specifically a unified energy-based audio-visual framework and a method for text saliency computation, for the detection of perceptually salient events from videos. Additionally, a movie summarization system for the automatic production of summaries is presented. Two kinds of evaluation were performed, an objective based on the saliency annotation of the database and an extensive qualitative human evaluation of the automatically produced summaries, where we investigated what composes high-quality movie summaries, where both methods verified the appropriateness of the proposed methods. The annotation of the database and the code for the summarization system can be found at http://cognimuse.cs.ntua.gr/database.

Movie summarization using bullet screen comments

Automatic movie summarization helps users to skim a movie in an efficient way. However, it is challenging because it requires the computer to automatically understand the movie content and the users’ opinions. Most previous works rely on the movie data itself without considering the opinions of the audience. In this paper, a novel approach for automatic movie summarization is presented by exploring a new type of user-generated data, i.e. bullet screen comments, which allow the audience to comment on the movie in a real-time manner. The number of the comments on a movie segment shows the exciting degree of the audience, while the content of the comments includes the concepts (e.g. the characters and the scenes) that interest the audience. In our approach, given a movie, bullet screen comments are utilized to select candidate highlight segments which are the most commented. Then the candidates are scored based on the number and the content of the bullet screen comments. Visual diversity is also considered in the scoring process. Finally, a subset of candidates which achieves the highest score is selected to compose a summary. Our experiments carried out on movies of different genres have shown the effectiveness of our proposed approach.

Exploiting character networks for movie summarization

Movie summarization focuses on providing as much information as possible for shorter movie clips while still keeping the content of the original movie and presenting a faster way for the audience to understand the movie. In this paper, we propose a novel method to summarize a movie based on character network analysis and the appearance of protagonist and main characters in the movie. Experiments were carried out for 2 movies (Titanic (1997) and Frozen (2013)) to show that our method outperforms conventional approaches in terms of the movie summarization rate.

Semantic movie summarization based on string of IE-RoleNets

Roles, their emotion, and interactions between them are three key elements for semantic content understanding of movies. In this paper, we proposed a novel movie summarization method to capture the semantic content in movies based on a string of IE-RoleNets. An IE-RoleNet (interaction and emotion rolenet) models the emotion and interactions of roles in a shot of the movie. The whole movie is represented as a string of IE-RoleNets. Summarization of a movie is transformed into finding an optimal substring with user-specified summarization ratio. Hierarchical substring mining is conducted to find an optimal substring of the whole movie. We have conducted objective and subjective experiments on our method. Experimental results show the ability of our method to capture the semantic content of movies.

What do you wish to see? A summarization system for movies based on user preferences

Video summarization aims at producing a compact version of a full-length video while preserving the significant content of the original video. Movie summarization condenses a full-length movie into a summary that still retains the most significant and interesting content of the original movie. In the past, several movie summarization systems have been proposed to generate a movie summary based on low-level video features such as color, motion, texture, etc. However, a generic summary, which is common to everyone and is produced based only on low-level video features will not satisfy every user. As users’ preferences for the summary differ vastly for the same movie, there is a need for a personalized movie summarization system nowadays. To address this demand, this paper proposes a novel system to generate semantically meaningful video summaries for the same movie, which are tailored to the preferences and interests of a user. For a given movie, shots and scenes are automatically detected and their high-level features are semi-automatically annotated. Preferences over high-level movie features are explicitly collected from the user using a query interface. The user preferences are generated by means of a stored-query. Movie summaries are generated at shot level and scene level, where shots or scenes are selected for summary skim based on the similarity measured between shots and scenes, and the user’s preferences. The proposed movie summarization system is evaluated subjectively using a sample of 20 subjects with eight movies in the English language. The quality of the generated summaries is assessed by informativeness, enjoyability, relevance, and acceptance metrics and Quality of Perception measures. Further, the usability of the proposed summarization system is subjectively evaluated by conducting a questionnaire survey. The experimental results on the performance of the proposed movie summarization approach show the potential of the proposed system.

Scene-Based Movie Summarization Via Role-Community Networks

Video summarization techniques aim at condensing a full-length video to a significantly shortened version that still preserves the major semantic content of the original video. Movie summarization, being a special class of video summarization, is particularly challenging since a large variety of movie scenarios and film styles complicate the problem. In this paper, we propose a two-stage scene-based movie summarization method based on mining the relationship between role-communities since the role-communities in earlier scenes are usually used to develop the role relationship in later scenes. In the analysis stage, we construct a social network to characterize the interactions between role-communities. As a result, the social power of each role-community is evaluated by the community's centrality value and the role communities are clustered into relevant groups based on the centrality values. In the summarization stage, a set of feasible summary combinations of scenes is identified and an information-rich summary is selected from these candidates based on social power preservation. Our evaluation results show that in at most test cases the proposed method achieves better subjective performance than attention-based and role-based summarization methods in terms of semantic content preservation for a movie summary.

Multimodal Saliency and Fusion for Movie Summarization Based on Aural, Visual, and Textual Attention

Multimodal streams of sensory information are naturally parsed and integrated by humans using signal-level feature extraction and higher level cognitive processes. Detection of attention-invoking audiovisual segments is formulated in this work on the basis of saliency models for the audio, visual, and textual information conveyed in a video stream. Aural or auditory saliency is assessed by cues that quantify multifrequency waveform modulations, extracted through nonlinear operators and energy tracking. Visual saliency is measured through a spatiotemporal attention model driven by intensity, color, and orientation. Textual or linguistic saliency is extracted from part-of-speech tagging on the subtitles information available with most movie distributions. The individual saliency streams, obtained from modality-depended cues, are integrated in a multimodal saliency curve, modeling the time-varying perceptual importance of the composite video stream and signifying prevailing sensory events. The multimodal saliency representation forms the basis of a generic, bottom-up video summarization algorithm. Different fusion schemes are evaluated on a movie database of multimodal saliency annotations with comparative results provided across modalities. The produced summaries, based on low-level features and content-independent fusion and selection, are of subjectively high aesthetic and informative quality.