Representation Of Multimedia Data Research Articles

A review of disentangled representation learning for visual data processing and analysis

表示学习是机器学习研究的核心问题之一。机器学习算法的输入表征从过去主流的手工特征过渡到现在面向多媒体数据的潜在表示，使算法性能获得了巨大提升。然而，视觉数据的表示通常是高度耦合的，即输入数据的所有信息成分被编码进同一个特征空间，从而互相影响且难以区分，使得表示的可解释性不高。解耦表示学习旨在学习一种低维的可解释性抽象表示，可以识别并分离出隐藏在高维观测数据中的不同潜在变化因素。通过解耦表示学习，可以捕获到单个变化因素信息并通过相对应的潜在子空间进行控制，因此解耦表示更具有可解释性。解耦表征可用于提高样本效率和对无关干扰因素的容忍度，为数据中的复杂变化提供一种鲁棒性表示，提取的语义信息对识别分类、域适应等人工智能下游任务具有重要意义。本文首先介绍并分析解耦表示的研究现状及其因果机制，总结解耦表示的3个重要性质。然后，将解耦表示学习算法分为4类，并从数学描述、类型特点及适用范围3个方面进行归纳及对比。随后，分类总结了现有解耦表示工作中常用的损失函数、数据集及客观评估指标。最后，总结了解耦表示学习在实际问题中的各类应用，并对其未来发展进行了探讨。;Representation learning is essential for machine learning technique nowadays. The transition of input representations have been developing intensively in algorithm performance benefited from the growth of hand-crafted features to the representation for multi-media data. However，the representations of visual data are often highly entangled. The interpretation challenges are to be faced because all information components are encoded into the same feature space. Disentangled representation learning（DRL）aims to learn a low-dimensional interpretable abstract representation that can sort the multiple factors of variation out in high-dimensional observations. In the disentangled representation，we can capture and manipulate the information of a single factor of variation through the corresponding latent subspace，which makes it more interpretable. DRL can improve sample efficiency and tolerance to the nuisance variables and offer robust representation of complex variations. Their semantic information is extracted and beneficial for artificial intelligence（AI）downstream tasks like recognition，classification and domain adaptation. Our summary is focused on brief introduction to the definition， research development and applications of DRL. Some of independent component analysis（ICA）-nonlinear DRL researches are covered as well since the DRL is similar to the identifiability issue of nonlinear independent component analysis（nonlinear ICA）. The cause and effects mechanism of DRL as high-dimensional ground truth data is generated by a set of unobserved changing factors（generating factors）. The DRL can be used to model the factors of variation in terms of latent representation，and the observed data generation process is restored. We summarize the key elements that a well-defined disentangled representation should be qualified into three aspects，which are 1）modularity，2）compactness，and 3）explicitness. First，explicitness consists of the two sub-requirements of completeness and informativeness. Then，current DRL types are categorized into 1）dimension-wise disentanglement，2）semantic-based disentanglement，3）hierarchical disentanglement，and 4） nonlinear ICA four types in terms of its formulation，characteristics，and scope of application. Dimension-wise disentanglement is assumed that the generative factors are solely and each dimension of latent vector can be separated and mapped，which is suitable for learning the disentangled representation of simple synthetic visual data. Semantic-based disentanglement is hypnotized that some semantic information is solely as well. The generative factors are group-disentangled in terms of specific semantics and they are mapped to different latent spaces，which is suitable for complicated ground truth data. Hierarchical disentanglement is based on the assumption that there is a correlation between generative factors at different levels of abstraction. The generative factors are disentangled by group from the bottom up and they can be mapped to latent space of different semantic abstraction levels to form a hierarchical disentangled representation. Nonlinear ICA provides an identifiable method for observed data-mixed disentangling unknown generative factors through a nonlinear reversible generator. For the motivation of loss functions，the loss functions can be commonly used in disentangled representation learning，which are grouped into three categories：1）modularity constraint：a single latent variable-constrained in the disentangled representation to capture only a single or a single group of factors of variation，and it promotes the separation of factors of variation mutually；2）explicitness constraint：current latent variable of the latent representation is activated to encode the ground truth of the corresponding generating factor effectively，and the entire latent representation contains complete information about all generative factors；and 3）multi-purpose constraint：lossrelated can optimize multiple disentangled representation，including modularity，compactness，and explicitness of the disentangled representation at the same time. The model-relevant can combine multiple loss constraint terms to form the final hybrid objective function. We compare the scope of application and limitations of each type of loss functions and summarize the classical disentangled representation works using the hybrid objective function further.

Multimedia data representation learning and applications

Multimedia data representation learning and applications

A Tensor-Based Framework for Software-Defined Cloud Data Center

Multimedia has been exponentially increasing as the biggest big data, which consist of video clips, images, and audio files. Processing and analyzing them on a cloud data center have become a preferred solution that can utilize the large pool of cloud resources to address the problems caused by the tremendous amount of unstructured multimedia data. However, there exist many challenges in processing multimedia big data on a cloud data center, such as multimedia data representation approach, an efficient networking model, and an estimation method for traffic patterns. The primary purpose of this article is to develop a novel tensor-based software-defined networking model on a cloud data center for multimedia big-data computation and communication. First, an overview of the proposed framework is provided, in which the functions of the representative modules are briefly illustrated. Then, three models,—forwarding tensor, control tensor, and transition tensor—are proposed for management of networking devices and prediction of network traffic patterns. Finally, two algorithms about single-mode and multimode tensor eigen-decomposition are developed, and the incremental method is employed for efficiently updating the generated eigen-vector and eigen-tensor. Experimental results reveal that the proposed framework is feasible and efficient to handle multimedia big data on a cloud data center.

A Multimedia Retrieval Framework Based on Semi-Supervised Ranking and Relevance Feedback

We present a new framework for multimedia content analysis and retrieval which consists of two independent algorithms. First, we propose a new semi-supervised algorithm called ranking with Local Regression and Global Alignment (LRGA) to learn a robust Laplacian matrix for data ranking. In LRGA, for each data point, a local linear regression model is used to predict the ranking scores of its neighboring points. A unified objective function is then proposed to globally align the local models from all the data points so that an optimal ranking score can be assigned to each data point. Second, we propose a semi-supervised long-term Relevance Feedback (RF) algorithm to refine the multimedia data representation. The proposed long-term RF algorithm utilizes both the multimedia data distribution in multimedia feature space and the history RF information provided by users. A trace ratio optimization problem is then formulated and solved by an efficient algorithm. The algorithms have been applied to several content-based multimedia retrieval applications, including cross-media retrieval, image retrieval, and 3D motion/pose data retrieval. Comprehensive experiments on four data sets have demonstrated its advantages in precision, robustness, scalability, and computational efficiency.

Multimedia data mining: state of the art and challenges

Advances in multimedia data acquisition and storage technology have led to the growth of very large multimedia databases. Analyzing this huge amount of multimedia data to discover useful knowledge is a challenging problem. This challenge has opened the opportunity for research in Multimedia Data Mining (MDM). Multimedia data mining can be defined as the process of finding interesting patterns from media data such as audio, video, image and text that are not ordinarily accessible by basic queries and associated results. The motivation for doing MDM is to use the discovered patterns to improve decision making. MDM has therefore attracted significant research efforts in developing methods and tools to organize, manage, search and perform domain specific tasks for data from domains such as surveillance, meetings, broadcast news, sports, archives, movies, medical data, as well as personal and online media collections. This paper presents a survey on the problems and solutions in Multimedia Data Mining, approached from the following angles: feature extraction, transformation and representation techniques, data mining techniques, and current multimedia data mining systems in various application domains. We discuss main aspects of feature extraction, transformation and representation techniques. These aspects are: level of feature extraction, feature fusion, features synchronization, feature correlation discovery and accurate representation of multimedia data. Comparison of MDM techniques with state of the art video processing, audio processing and image processing techniques is also provided. Similarly, we compare MDM techniques with the state of the art data mining techniques involving clustering, classification, sequence pattern mining, association rule mining and visualization. We review current multimedia data mining systems in detail, grouping them according to problem formulations and approaches. The review includes supervised and unsupervised discovery of events and actions from one or more continuous sequences. We also do a detailed analysis to understand what has been achieved and what are the remaining gaps where future research efforts could be focussed. We then conclude this survey with a look at open research directions.

XFlavor: providing XML features in media representation

We present XFlavor, a framework for providing XML representation of multimedia data. XFlavor can be used to convert multimedia data back and forth between binary and XML representations. Compared to bitstreams, XML documents are easier to access and manipulate, and consequently, the development of multimedia processing software is greatly facilitated, as one generic XML parser can be used to read and write different types of data in XML form.

A simulated annealing approach for multimedia data placement

Multimedia applications are characterized by their strong timing requirements and constraints and thus multimedia data storage is a critical issue in the overall system's performance and functionality. This paper describes multimedia data representation models that effectively guide data placement towards the improvement of the Quality of Presentation for the considered multimedia applications. The performance of both constructive placement and iterative improvement placement algorithms is evaluated and discussed. Emphasis is given on placement schemes which are based on the simulated annealing optimization algorithm. A placement policy, based on a self-improving version of the simulated annealing (SISA) algorithm is applied and evaluated. Performance of the placement policies is experimentally evaluated on a simulated tertiary storage subsystem. As proven by the experimentation, the proposed approach shows considerable gain in terms of seek and service times. The improvements of the proposed SISA approach are in the range of 40% when compared to random placement and at the range of 15–35% when compared to the typical simulated annealing algorithm, depending a lot on the initial configuration and the neighborhood search.

Hierarchical data placement for navigational multimedia applications

Navigation has added interactivity in nowadays multimedia applications which support effective accessing to objects of various formats and presentation requirements. Storage issues need to be reconsidered for the new type of navigational multimedia applications, in order to improve system’s performance. This paper addresses the problem of multimedia data storage towards improving data accessibility and request servicing under navigational applications. A navigational graph-based model (for the multimedia data representation) is proposed to guide the data placement under a hierarchical storage topology. The multimedia data dependencies, access frequencies and timing constraints are used to characterize the graph nodes which correspond to multimedia objects allocated at the tertiary storage level. Based on certain defined popularity criteria, data are elevated and placed on secondary level towards improving both the request servicing and data accessibility. The proposed multimedia data elevation is a prefetching approach since it is performed “apriori” (not on demand) based on previously extracted user access patterns. Appropriate data placement policies are also employed at the secondary level, and a simulation model has been developed based on current commercial tertiary and secondary storage devices. This model is used to evaluate the proposed popularity based data elevation approach as employed under a hierarchical storage subsystem. Experimentation is performed under artificial data workloads and it is shown that the proposed hierarchical data placement approach considerably improves data accessing and request servicing in navigational multimedia applications. The iterative improvement placement is proven to outperform earlier related multimedia data placement policies with respect to commonly used performance metrics.

An Approach to a Content-Based Retrieval ofMultimedia Data

This paper presents a data model tailored for multimedia data representation, along with the main characteristics of a Multimedia Query Language that exploits the features of the proposed model. The model addresses data presentation, manipulation and content-based retrieval. It consists of three parts: a Multimedia Description Model, which provides a structural view of raw multimedia data, a Multimedia Presentation Model, and a Multimedia Interpretation Model which allows semantic information to be associated with multimedia data. The paper focuses on the structuring of a multimedia data model which provides support for content-based retrieval of multimedia data. The Query Language is an extension of a traditional query language which allows restrictions to be expressed on features, concepts, and the structural aspects of the objects of multimedia data and the formulation of queries with imprecise conditions. The result of a query is an approximate set of database objects which partially match such a query.