Visual Summarization Research Articles

EventThread: Visual Summarization and Stage Analysis of Event Sequence Data.

Event sequence data such as electronic health records, a person's academic records, or car service records, are ordered series of events which have occurred over a period of time. Analyzing collections of event sequences can reveal common or semantically important sequential patterns. For example, event sequence analysis might reveal frequently used care plans for treating a disease, typical publishing patterns of professors, and the patterns of service that result in a well-maintained car. It is challenging, however, to visually explore large numbers of event sequences, or sequences with large numbers of event types. Existing methods focus on extracting explicitly matching patterns of events using statistical analysis to create stages of event progression over time. However, these methods fail to capture latent clusters of similar but not identical evolutions of event sequences. In this paper, we introduce a novel visualization system named EventThread which clusters event sequences into threads based on tensor analysis and visualizes the latent stage categories and evolution patterns by interactively grouping the threads by similarity into time-specific clusters. We demonstrate the effectiveness of EventThread through usage scenarios in three different application domains and via interviews with an expert user.

Molecular Surface Maps.

We present Molecular Surface Maps, a novel, view-independent, and concise representation for molecular surfaces. It transfers the well-known world map metaphor to molecular visualization. Our application maps the complex molecular surface to a simple 2D representation through a spherical intermediate, the Molecular Surface Globe. The Molecular Surface Map concisely shows arbitrary attributes of the original molecular surface, such as biochemical properties or geometrical features. This results in an intuitive overview, which allows researchers to assess all molecular surface attributes at a glance. Our representation can be used as a visual summarization of a molecule's interface with its environment. In particular, Molecular Surface Maps simplify the analysis and comparison of different data sets or points in time. Furthermore, the map representation can be used in a Space-time Cube to analyze time-dependent data from molecular simulations without the need for animation. We show the feasibility of Molecular Surface Maps for different typical analysis tasks of biomolecular data.

ETGraph: A graph-based approach for visual analytics of eye-tracking data

Mind wander(ing) (MW) or zoning out is a ubiquitous phenomenon where attention involuntary shifts from task-related processing to task-unrelated thoughts. Unfortunately, MW is a highly internal state so it cannot be readily inferred from overt behaviors and expressions. To help experts investigate mind wanderings, we present a graph-based approach for visual analytics of eye-tracking data, which utilizes the graph representations to illustrate the reading patterns and further help experts detect and verify mind wanderings based on the graph structures and other graph attributes. The input data are collected from multiple participants reading multiple pages of a book on a computer screen. Our approach first clusters fixations into fixation clusters, then creates the eye-tracking graph, i.e., ETGraph, for use in conjunction with the standard page view, time view, and statistics view. The graph view presents a visual representation of the actual reading patterns of a single participant or multiple participants and therefore serves as the main visual interface for exploration and navigation. We design a suite of techniques to help users identify common reading patterns and outliers for analytical reasoning at three different levels of detail: single participant single page, single participant multiple pages, and multiple participants single page. Interactive querying and filtering functions are provided for reducing visual clutter in the visualization and enabling users to answer questions and glean insights. Our tool also facilitates the detection and verification of mind wandering that the experts seek to investigate. We conduct a user study and an expert evaluation to assess the effectiveness of ETGraph in terms of its visual summarization and comparison capabilities.

Homophily in coauthorship networks of East European sociologists.

We study to what degree and how homophily and network properties affect individual citation counts of researchers in the sociology departments of three East European countries, namely Poland, Romania, and Slovenia. We built first-order personal coauthorship networks out of the Web of Science publication records. Each sociologist is assigned as a focal node or ego, while her coauthors are alters. We analyze the data using structural measurements methods, hierarchical regression models, and we make visualizations based on the clustered graph technique. For all three populations, our results indicate that the mean score of the citations of alters substantially predicts the citation counts of egos. In particular, citation similarity increases the chances for coauthorship ties. Evidence for the impact of network properties on the citation levels of egos is mixed. For Poland, normalized ego-betweenness shows a negative effect on citation counts, while network density displays a positive one. For Romania and Slovenia, network characteristics have only a minor impact. Even if the visual summarization of the personal networks uncovers a wide palette of coauthorship patterns, homophily appears to be pervasive. These results are relevant for domestic policy makers who aim to improve the aggregated research performance in East European countries.

Visual summarization of image collections by fast RANSAC

In this paper we propose a novel approach to select a summary set of images from a large image collection by improved Random Sample Consensus (RANSAC) and Affinity Propagation (AP) clustering. It can automatically select a small set of representatives to highlight all the significant visual properties of a given image collection. The proposed framework mainly composes four stages. First, the scale-invariant feature of each image is extracted by Scale Invariant Feature Transform (SIFT). Second, keypoints of two images are matched and ranked based on nearest neighbor ratio. The representative dataset of RANSAC is established by a minimal number of optimal matches. Third, the target homographic matrix is fitted based on the representative dataset. Mismatches are filtered out via the homographic matrix. Finally, summarization is automatically formulated as an optimization framework by AP clustering. We conduct experiments on a set of Paris which is consisting of 1000 images downloaded from Flickr. The results show that the proposed approach significantly outperforms other methods.

Visualizing Complex Data With Embedded Plots

This article describes a class of graphs, embedded plots, that are particularly useful for analyzing large and complex datasets. Embedded plots organize a collection of graphs into a larger graphic, which can display more complex relationships than would otherwise be possible. This arrangement provides additional axes, prevents overplotting, and allows for multiple levels of visual summarization. Embedded plots also preprocess complex data into a form suitable for the human cognitive system, which can facilitate comprehension. We illustrate the usefulness of embedded plots with a case study, discuss the practical and cognitive advantages of embedded plots, and demonstrate how to implement embedded plots as a general class within visualization software, something currently unavailable. This article has supplementary material online.

Designing a Ranking System for Product Search Engine Based on Mining UGC

Designing a Ranking System for Product Search Engine Based on Mining UGC Dr. Samira Chaabna, Professor Wang Hu, Dr. Mohammed Lutf Abstract With the spread of e-commerce platforms, it becomes extremely difficult for the consumer to choose the right product from a large number of identical products, and different sellers based only on his/her own experience, product pictures or product metadata. Customers reviews present a rich and valuable source of information for potential consumers and manufacturers which can be very helpful in the decision process, but reading all of the available reviews is a hard task and time-consuming. Thus, the automated mining of these reviews and extracting product features in order to generate a raking system present a valuable and useful tool for consumers to make a well-informed decision. In this paper, we propose a ranking system for product search engine based on mining customers reviews. The product features pairs are extracted using Stanford typed dependencies. The evaluation process of the proposed system has passed through two levels: first measuring of accuracy of product feature extraction and classification, and the second one is the determination of the efficiency of the search results and the usability of visual summarization. The results show a high level of accuracy in feature/opinion pairs extraction and high level of participant satisfaction with the ranking and the summarization. Full Text: PDF DOI: 10.15640/jmise.v2n1a2

Effective browsing of image search results via diversified visual summarization by clustering and refining clusters

The unprecedented growth in applications making use of digital images and multimedia references raised the requirement for image and topic search. Systematic processing of this information is a basic prerequisite for effective analysis of this information, as well as organization and management of it. Likewise, large collections of images are available on the Web, and many search engines provide the possibility of Web image searching based on the keywords. However, there are some problems in finding the desirable image according to the user needs. These problems include inexpressiveness of queries in description of user requirement, large number of irrelevant images to the intended search, lack of summarization, time-consuming review of overall images and lack of diversity. Clustering of image search results can be an efficient solution in solving these problems. In this work, several Folding-based algorithms have been proposed for clustering of image search results. In these algorithms, the efficiency, in comparison with Folding algorithm, is improved through a more effective selection of cluster’s representatives, fuzziness, weight and utilization of hierarchical algorithm preferences. In the represented system, in order to fit clusters with data more appropriately, an algorithm is proposed for refining the clusters. The proposed clustering causes a more convenient task in retrieval process for the user and also causes the efficient retrieval of images. According to the experiences, the proposed method improves the acceptable precision of image clustering.

Perception of Average Value in Multiclass Scatterplots

The visual system can make highly efficient aggregate judgements about a set of objects, with speed roughly independent of the number of objects considered. While there is a rich literature on these mechanisms and their ramifications for visual summarization tasks, this prior work rarely considers more complex tasks requiring multiple judgements over long periods of time, and has not considered certain critical aggregation types, such as the localization of the mean value of a set of points. In this paper, we explore these questions using a common visualization task as a case study: relative mean value judgements within multi-class scatterplots. We describe how the perception literature provides a set of expected constraints on the task, and evaluate these predictions with a large-scale perceptual study with crowd-sourced participants. Judgements are no harder when each set contains more points, redundant and conflicting encodings, as well as additional sets, do not strongly affect performance, and judgements are harder when using less salient encodings. These results have concrete ramifications for the design of scatterplots.

Learning Crowdsourced User Preferences for Visual Summarization of Image Collections

In this paper we propose a novel approach to selecting images suitable for inclusion in the visual summaries. The approach is grounded in insights about how people summarize image collections. We utilize the Amazon Mechanical Turk crowdsourcing platform to obtain a large number of manually created visual summaries as well as information about criteria for image inclusion in the summary. Based on these large-scale user tests, we propose an automatic image selection approach, which jointly utilizes the analysis of image content, context, popularity, visual aesthetic appeal as well as the sentiment derived from the comments posted on the images. In our approach we do not describe images based on their properties only, but also in the context of semantically related images, which improves robustness and effectively enables propagation of sentiment, aesthetic appeal as well as various inherent attributes associated with a particular group of images. We discuss the phenomenon of a low inter-user agreement, which makes an automatic evaluation of visual summaries a challenging task and propose a solution inspired by the text summarization and machine translation communities. The experiments performed on a collection of geo-referenced Flickr images demonstrate the effectiveness of our image selection approach.

Generating Visual Summaries of Geographic Areas Using Community-Contributed Images

In this paper, we present a novel approach for automatic visual summarization of a geographic area that exploits user-contributed images and related explicit and implicit metadata collected from popular content-sharing websites. By means of this approach, we search for a limited number of representative but diverse images to represent the area within a certain radius around a specific location. Our approach is based on the random walk with restarts over a graph that models relations between images, visual features extracted from them, associated text, as well as the information on the uploader and commentators. In addition to introducing a novel edge weighting mechanism, we propose in this paper a simple but effective scheme for selecting the most representative and diverse set of images based on the information derived from the graph. We also present a novel evaluation protocol, which does not require input of human annotators, but only exploits the geographical coordinates accompanying the images in order to reflect conditions on image sets that must necessarily be fulfilled in order for users to find them representative and diverse. Experiments performed on a collection of Flickr images, captured around 207 locations in Paris, demonstrate the effectiveness of our approach.

Interactive multimedia computing

In recent years, we have witnessed the flourish of multimedia data on the Internet. To facilitate humans in accessing and managing the explosively growing multimedia contents, extensive research efforts have been dedicated to automatic multimedia analysis and processing in the past decades, such as categorization, annotation and indexing. However, despite great advances achieved, several key difficulties still exist, such as the well-known semantic gap in multimedia modeling. It is evident from recent results that, without additional information resources, most of the semantic gap problems can hardly be solved automatically within the near future. On the other hand, we have witnessed the power of collective human efforts in the Web 2.0 era in providing high-quality tags and comments to large amounts of images and videos in sites such as Flickr and YouTube. In fact, a lot more can be accomplished through simple online games such as the ESP. Hence, more and more researchers believe that a possible approach to addressing the semantic gap problem is to incorporate the efforts of humans into the computational process, i.e., by combining human intelligence and automated computer processing to jointly tackle the problems in a collaborative manner. The past decade has witnessed the increase of such efforts, such as relevance feedback in content-based image retrieval, active learning in multimedia modeling, the interactive video search evaluation task in TRECVID, new search and browsing interfaces in VideoOlympics to facilitate humans’ interaction, and the recent human computation efforts such as the ESP game on Google image search website. This special issue is organized with the purpose of introducing novel research work on interactive multimedia computing. Submissions have come from an open call for paper. With the assistance of dedicated referees, five papers have been selected after two rounds of rigorous reviews. These papers cover widely subtopics of interactive multimedia computing, including game-based image annotation, interactive TV, interactive cartoon synthesis, and so on. In the first paper ‘‘Adding Semantics to Image Region Annotations with the Name-It-Game’’, Steggink and Snoek introduce a system that accomplishes region-level image annotation with a game. It establishes a set of keywords that describe objects by exploring WordNet, and the keywords are assigned to image regions with a two-player ‘‘reveal and guess’’ game. They also explore WordNet to address the word ambiguity problem. In addition to introducing the system, another contribution of the paper is its review of existing manual image annotation techniques, in particular the comprehensive study of game-based annotation. In the second paper ‘‘Interactive Browsing via Diversified Visual Summarization for Image Search Results’’, Wang et al. introduce a scheme for the summarization and browsing of image search results. It adopts a dynamic absorbing random walk approach to summarize the image search results. The summarization is visualized on a 2D panel and users’ browsing is facilitated with dynamic scale change and a browsing path tracking tool. Experiments with a set of diverse queries have demonstrated the effectiveness of the approach. The third paper, ‘‘Security and Privacy Requirements in Interactive TV’’, discusses the security and privacy issues in the context of interactive TV. It introduces an interactive M. Wang (&) J. Tang T.-S. Chua National University of Singapore, Singapore, Singapore e-mail: eric.mengwang@gmail.com

MI-MERCURY: A mobile agent architecture for ubiquitous retrieval and delivery of multimedia information

Mining multimedia information in the Web is in general an arduous task, due to the fact that, (a) humans perceive media content using high level concepts, (b) the subjective and vagueness of content interpretation, and (c) the fact that relevant data are often hidden in a huge amount of irrelevant information. In addition, delivering and distributing the retrieved information to a wide range of terminal devices of different properties over a wide range of networks to users of different preferences requires new tools and mechanisms for content transformation and adaptation. Other problems concern the language that the data are stored, which may not be the user's preferred language. To address these issues we propose an integrated, reconfigurable, adaptable and open architecture for mining, indexing and retrieving multimedia information based on a mobile agent technology scheme. The proposed architecture consists of three integral subsystems: the acquisition module, responsible for searching and retrieving media data (both textual and visual), the transformation module, able to adapt and transform the mined information to other forms of representation, and the distribution module for delivering and adapting the retrieved data in terms of terminal devices, network channels and user's preferences. The system is based on a reconfigurable architecture which is able to dynamically and automatically update the system response to user's actual needs and preferences, by extending descriptor classes that are considered more relevant by the users. New innovative algorithms are presented in this paper both at each system module as well as in the system integration. The system supports efficient content adaptation mechanisms, textual and visual summarization (both sequential and hierarchical), automatic language translation, ontological representation, visual processing and web-based data mining. Experimental analysis on real-life web sites has been performed to test the efficiency of the proposed scheme and compare it with other approaches presented in the literature.

Web-based Digital Video Sequencing System

In order to access and retrieve Internet shared videos, there is a clear need for structuring to decrease search and loading time. Video segmentation is the process of dividing a video clip into segments of similar characteristics. Based on our segmentation research, this study proposes a complete Digital Video Sequencing and retrieval tool, aimed for the Internet users. This ongoing research aspires to provide a fast and efficient visual summarization and searching of video content such that users can efficiently determine whether they are interested in the video before they have to download it from the Internet. This system consists of three major modules. The first module is responsible for automatic segmentation of video scenes. The second module structures the video as a Contents Structure (VCS) ready for lookup. Finally, the last module provides the users with video display Interface Program. Our experiments on a number of various videos demonstrate the efficiency of this system.

Content-based access to video objects: Temporal Segmentation, visual summarization, and feature extraction

The classical approach to content-based video access has been ‘frame-based’, consisting of shot boundary detection, followed by selection of key frames that characterize the visual content of each shot, and then clustering of the camera shots to form story units. However, in an object-based multimedia environment, content-based random access to individual video objects becomes a desirable feature. To this effect, this paper introduces an ‘object-based’ approach to temporal video partitioning and content-based indexing, where the basic indexing unit is ‘lifespan of a video object’, rather than a ‘camera shot’ or a ‘story unit’. We propose to represent each video object by an adaptive 2D triangular mesh. A mesh-based object tracking scheme is then employed to compute the motion trajectories of all mesh node points until the object exits the field of view. A new similarity measure that is based on motion discontinuities and shape changes of the tracked object is defined to detect content changes, resulting in temporal lifespan segments. A set of ‘key snapshots’ which constitute a visual summary of the lifespan of the object is automatically selected. These key snapshots are then used to animate objects of interest using tracked motion trajectories for a moving visual representation. The proposed scheme provides such functionalities as object-based search/browsing for interactive video retrieval, surveillance video analysis, and object-based content manipulation/editing for studio postprocessing and desktop multimedia authoring. The approach is applicable to any video data where the initial appearance of object(s) can be specified, and the object motion can be modeled by a piecewise affine transformation. The system is demonstrated using different types of video: virtual studio productions (composited video), surveillance video, and TV broadcast video.