Multimedia Community Research Articles

A Novel Authorization Scheme for Multimedia Social Networks Under Cloud Storage Method by Using MA-CP-ABE

This article finds that with the development of network technology and the change of a users' social habits, the existing multimedia social network, whether it is one-to-one authorization or one-to-many authorization, are having difficulty meeting the needs of users and cloud storage service. In order to solve this problem, this article proposes a new authorization scheme of multimedia community, based on attribute encryption. Use MA-CP-ABE solution to enhance system service efficiency. This program contributes to two aspects: (1) combining user trust with attribute-based authorizations, and (2) enhancing the flexibility and security of the solution. The scheme has been applied in the prototype system and achieved good results. Results prove that the program has good application value for cloud storage technology, IoT and many other fields.

Check Out This Place: Inferring Ambiance From Airbnb Photos

Airbnb is changing the landscape of the hospitality industry, and to this day, little is known about the inferences that guests make about Airbnb listings. Our work constitutes a first attempt at understanding how potential Airbnb guests form first impressions from images, one of the main modalities featured on the platform. We contribute to the multimedia community by proposing the novel task of automatically predicting human impressions of ambiance from pictures of listings on Airbnb. We collected Airbnb images, focusing on the countries Switzerland and Mexico as case studies, and used crowdsourcing mechanisms to gather annotations on physical and ambiance attributes, finding that agreement among raters was high for most of the attributes. Our cluster analysis showed that both physical and psychological attributes could be grouped into three clusters. We then extracted state-of-the-art features from the images to automatically infer the annotated variables in a regression task. Results show the feasibility of predicting ambiance impressions of homes on Airbnb, with up to 42% of the variance explained by our model, and best results were obtained using activation layers of deep convolutional neural networks trained on the Places dataset, a collection of scene-centric images.

Affective image classification via semi-supervised learning from web images

Affective image classification has drawn increasing research attentions in the affective computing and multimedia communities. Despite many solutions proposed in the literature, it remains a major challenge to bridge the semantic gap between visual features of images and their affective characteristics, partly due to the lack of adequate training samples, which can be largely ascribed to the all-consuming nature of affective image annotation. In this paper, we propose a novel affective image classification algorithm based on semi-supervised learning from web images (SSL-WI). This algorithm consists of four major steps, including color and texture feature extraction, baseline classifier construction, feature selection, and jointly using training images and retrieved web images to re-train the classifier. We have applied this algorithm, the baseline classifier that is not trained by web images, and two state-of-the-art algorithms to differentiating color images in a three-dimensional discrete emotional space. Our results suggest that, with the scheme of semi-supervised learning from web images, the proposed algorithm is able to produce more accurate affective image classification than other three approaches.

Describing Video With Attention-Based Bidirectional LSTM.

Video captioning has been attracting broad research attention in the multimedia community. However, most existing approaches heavily rely on static visual information or partially capture the local temporal knowledge (e.g., within 16 frames), thus hardly describing motions accurately from a global view. In this paper, we propose a novel video captioning framework, which integrates bidirectional long-short term memory (BiLSTM) and a soft attention mechanism to generate better global representations for videos as well as enhance the recognition of lasting motions in videos. To generate video captions, we exploit another long-short term memory as a decoder to fully explore global contextual information. The benefits of our proposed method are two fold: 1) the BiLSTM structure comprehensively preserves global temporal and visual information and 2) the soft attention mechanism enables a language decoder to recognize and focus on principle targets from the complex content. We verify the effectiveness of our proposed video captioning framework on two widely used benchmarks, that is, microsoft video description corpus and MSR-video to text, and the experimental results demonstrate the superiority of the proposed approach compared to several state-of-the-art methods.

Image Captioning with Deep Bidirectional LSTMs and Multi-Task Learning

Generating a novel and descriptive caption of an image is drawing increasing interests in computer vision, natural language processing, and multimedia communities. In this work, we propose an end-to-end trainable deep bidirectional LSTM (Bi-LSTM (Long Short-Term Memory)) model to address the problem. By combining a deep convolutional neural network (CNN) and two separate LSTM networks, our model is capable of learning long-term visual-language interactions by making use of history and future context information at high-level semantic space. We also explore deep multimodal bidirectional models, in which we increase the depth of nonlinearity transition in different ways to learn hierarchical visual-language embeddings. Data augmentation techniques such as multi-crop, multi-scale, and vertical mirror are proposed to prevent overfitting in training deep models. To understand how our models “translate” image to sentence, we visualize and qualitatively analyze the evolution of Bi-LSTM internal states over time. The effectiveness and generality of proposed models are evaluated on four benchmark datasets: Flickr8K, Flickr30K, MSCOCO, and Pascal1K datasets. We demonstrate that Bi-LSTM models achieve highly competitive performance on both caption generation and image-sentence retrieval even without integrating an additional mechanism (e.g., object detection, attention model). Our experiments also prove that multi-task learning is beneficial to increase model generality and gain performance. We also demonstrate the performance of transfer learning of the Bi-LSTM model significantly outperforms previous methods on the Pascal1K dataset.

Sensing Technologies for Monitoring Serious Mental Illnesses

Mental health is an urgent global issue. Around 450 million people suffer from serious mental illnesses worldwide, which results in devastating personal outcomes and huge societal burden. Effective symptom monitoring and personalized interventions can significantly improve mental health care across different populations. However, traditional clinical methods often fall short when it comes to real-time monitoring of symptoms. Sensing technologies can address these issues by enabling granular tracking of behavioral, physiological, and social signals relevant to mental health. In this article, we describe how sensing technologies can be used to diagnose and monitor patient states for numerous serious mental illnesses. We also identify current limitations and potential future directions. We believe that the multimedia community can build on sensing technologies to enable efficient clinical decision-making in mental health care. Specifically, innovative multimedia systems can help identify and visualize personalized early-warning signs from complex multimodal signals, which could lead to effective intervention strategies and better preemptive care.

SENSIBILITAS MULTIMEDIA DAN KETERKAITANNYA DENGAN KESULITAN DALAM PEMBELAJARAN BAHASA JEPANG

(Title: Multimedia Sensibility And Its Relation With Difficulties In Japanese Language Learning) The development of technology and the popularity of multimedia as a means of conveying information have conditioned society to change into a multimedia community with unique characteristics, sensibility, and needs. This study aims to see the effect of multimedia sensibility on the difficulties in the process of learning Japanese. Data were taken from a survey conducted on the students and analyzed using concepts related to the characteristics of the multimedia society. The results of this study show that multimedia sensibility has an effect on student’s difficulties in understanding learning materials delivered using different media. The new needs of the multimedia society have to be answered with various adjustments. However, by only paying attention to the multimedia sensibility and needs, there is a possibility that learning activities become less reliable in sharpening analytical skills. Therefore, choosing the right materials and methods is important so that the learner’s interest can be sustained by adjusting to their needs and sensibility, and also considering activities that can sharpen other abilities, such as analytical skills, which is very important for student self- development.

Opinion column

As promised in our introductory edition, this column will report highlights and lowlights of online discussion threads among the members of the Multimedia community (see our Facebook MM Community Discussion group).

Impact of the new @sigmm records

The SIGMM Records have renewed, with the ambition of continue being a useful resource for the multimedia community. The intention is to provide a forum for (open) discussion and to become a primary source of information (and of inspiration!).

An extensive performance evaluation of full-reference HDR image quality metrics

High dynamic range (HDR) image and video technology has recently attracted a great deal of attention in the multimedia community, as a mean to produce truly realistic video and further improve the quality of experience (QoE) of emerging multimedia services. In this context, measuring the quality of compressed HDR content plays a fundamental role. However, full-reference (FR) HDR visual quality assessment poses new challenges with respect to the conventional low dynamic range case. Quality metrics have to be redesigned or adapted to HDR, and understanding their reliability to predict users’ judgments is even more critical due to the still limited availability of HDR displays to perform subjective evaluations. The goal of this paper is to provide a complete and thorough survey of the performance of the most popular HDR FR image quality metrics. To this end, we gather several existing HDR image databases with subjective quality annotations, in addition to a new one created by ourselves. After aligning the scores in these databases, we obtain an extensive set of 690 compressed HDR images, along with their subjective quality. Next, we analyze in depth many FR metrics, including those used in MPEG standardization, using both classical correlation analyses and classification accuracy. We believe that our results could serve as the most complete and comprehensive benchmark of image quality metrics in the field of HDR image compression.

Automatic image annotation with real-world community contributed data set

With the massive explosion of social multimedia community, social images have become very popular in our daily life. The image-associated labels are a valuable resource for automatic image annotation, but they tend to be unreliable. In this paper, we exploit the problem of image annotation from real-world community contributed images and their associated incorrect, insufficient, and personalized labels. We present SNTag, a novel semantic neighborhood learning method, on which image annotation task can be efficiently carried out in real-world scenario. First, we propose to use image-associated labels as the supervising information to guide the replenishment of training images, which enable the labels for training image not only more sufficient, but also more correct. Then, the “semantic balanced neighborhood” for image is generated, thus enabling the presence of more rare labels in image label list. Furthermore, we generate “semantic consistent neighborhood” within corresponding “semantic balanced neighborhood”. The retrieved neighbor images are not only visually alike but also semantically related. Contrary to earlier work, these neighbors are retrieved from the same subspace by the integration of metric learning embedded in multiple labels and sparse reconstruction. Based on the neighbor set, we propose a novel algorithm to assign the optimal labels to the image, which is more robust to noise. We conduct extensive experiments on several standard real-world benchmark data sets downloaded from community websites. The experimental results demonstrate that it outperforms the current state-of-the-art methods.

Cross-Modal Hashing via Rank-Order Preserving

Due to the query effectiveness and efficiency, cross-modal similarity search based on hashing has acquired extensive attention in the multimedia community. Most existing methods do not explicitly employ the ranking information when learning hash functions, which is quite important for building practical retrieval systems. To solve this issue, this paper proposes a rank-order preserving hashing (RoPH) method with a novel regression-based rank-order preserving loss that has provable large margin property and is easy to optimize. Moreover, we jointly learn the binary codes and hash functions instead of using any relaxation trick. To solve the induced optimization problem, the alternating descent technique is adopted and each subproblem can be solved conveniently. Specifically, we show that the involved binary quadratic programming subproblem with respect to an introduced auxiliary binary variable satisfies submodularity, enabling us to use the off-the-shelf graph-cut algorithms to solve it exactly and efficiently. Extensive experiments on three benchmarks demonstrate that RoPH significantly improves the ranking quality over the state of the arts.

Neighborhood Matching for Image Retrieval

In the last few years, large-scale image retrieval has attracted a lot of attention from the multimedia community. Usual approaches addressing this task first generate an initial ranking of the reference images using fast approximations that do not take into consideration the spatial arrangement of local features in the image (e.g., the bag-of-words paradigm). The top positions of the rankings are then re-estimated with verification methods that deal with more complex information, such as the geometric layout of the image. This verification step allows pruning of many false positives at the expense of an increase in the computational complexity, which may prevent its application to large-scale retrieval problems. This paper describes a geometric method known as neighborhood matching (NM), which revisits the keypoint matching process by considering a neighborhood around each keypoint and improves the efficiency of a geometric verification step in the image search system. Multiple strategies are proposed and compared to incorporate NM into a large-scale image retrieval framework. A detailed analysis and comparison of these strategies and baseline methods have been investigated. The experiments show that the proposed method not only improves the computational efficiency, but also increases the retrieval performance and outperforms state-of-the-art methods in standard datasets, such as the Oxford 5 k and 105 k datasets , for which the spatial verification step has a significant impact on the system performance.

Content-based processing and analysis of endoscopic images and videos: A survey

In recent years, digital endoscopy has established as key technology for medical screenings and minimally invasive surgery. Since then, various research communities with manifold backgrounds have picked up on the idea of processing and automatically analyzing the inherently available video signal that is produced by the endoscopic camera. Proposed works mainly include image processing techniques, pattern recognition, machine learning methods and Computer Vision algorithms. While most contributions deal with real-time assistance at procedure time, the post-procedural processing of recorded videos is still in its infancy. Many post-processing problems are based on typical Multimedia methods like indexing, retrieval, summarization and video interaction, but have only been sparsely addressed so far for this domain. The goals of this survey are (1) to introduce this research field to a broader audience in the Multimedia community to stimulate further research, (2) to describe domain-specific characteristics of endoscopic videos that need to be addressed in a pre-processing step, and (3) to systematically bring together the very diverse research results for the first time to provide a broader overview of related research that is currently not perceived as belonging together.

FreeScup: A Novel Platform for Assisting Sculpture Pose Design

Sculpture design is challenging due to its inherent difficulty in characterizing artworks quantitatively; thus, few works have been done to assist sculpture design in the past decades in the multimedia community. We have cooperated with several sculptors on analyzing styles of different artists consisting of Giacometti, Augeuste Rodin, Henry Moore, and Marino Marini from which we find pose editing plays an important role in sculpture design. Motivated by this, we present a novel platform that allows sculptors to edit virtual three-dimensional (3-D) sculptures by a free way. The proposed platform consists of three modules, namely, sculpture initialization , sculptor-sculpture mapping , and interactive pose editing . In sculpture initialization, a virtual 3-D sculpture is first incrementally reconstructed from multiview images. Then, we define Laplace operator and its corresponding spectrum to describe the geometry information of the reconstructed sculpture. During sculptor–sculpture mapping, we apply spectral analysis on the low-frequency parts of the spectrum to search for candidate editing points on the surface of the sculpture. Next, body actions of the sculptor are captured by Kinect and further mapped onto editing points as a predefined configuration set. Finally, during interactive pose editing, a real-time Kinect-driven sculpture pose editing scheme is presented, which not only preserves geometry features of the sculpture but also allows instant changes of sculpture poses. We demonstrate that our platform successfully assists sculptors on real-time pose editing by comparing its performance with those of the existing sculpture assisting methods.

The Continuing Reinvention of Content-Based Retrieval: Multimedia Is Not Dead

In reviewing how content-based retrieval has evolved over the years, the authors argue that the next grand challenge for the multimedia community will be understanding and formally modeling the flow of life around us, over many modalities and scales. This will require supplementing generic signal/sensor-based retrieval with syntactical, semantic, and pragmatics-based approaches. To determine whether a data-driven approach supports their observations regarding the evolution of content-based retrieval, the authors examine the types of research reported at the ACM International Conference on Multimedia from 2005 to 2015.

Multimedia at CHI: Telepresence at Work for Remote Conference Participation

Susanne Boll recounts her experience as a telepresence robot attending this year’s ACM Conference on Human Factors in Computing Systems (CHI), illustrating the fusion of the state of the art in robotics, long-distance interaction, and shared remote-audio-visual experiences. As a researcher in human computer interaction, yet with an international team facing restricted travel owing to the newly imposed US travel ban, she opted for remote participation at CHI 2017. Furthermore, as a member of the multimedia community, she wanted to explore first-hand how multimedia is at work in the field of telepresence.

Extending multimedia languages to support multimodal user interactions

Historically, the Multimedia community research has focused on output modalities, through studies on timing and multimedia processing. The Multimodal Interaction community, on the other hand, has focused on user-generated modalities, through studies on Multimodal User Interfaces (MUI). In this paper, aiming to assist the development of multimedia applications with MUIs, we propose the integration of concepts from those two communities in a unique high-level programming framework. The framework integrates user modalities --both user-generated (e.g., speech, gestures) and user-consumed (e.g., audiovisual, haptic)-- in declarative programming languages for the specification of interactive multimedia applications. To illustrate our approach, we instantiate the framework in the NCL (Nested Context Language) multimedia language. NCL is the declarative language for developing interactive applications for Brazilian Digital TV and an ITU-T Recommendation for IPTV services. To help evaluate our approach, we discuss a usage scenario and implement it as an NCL application extended with the proposed multimodal features. Also, we compare the expressiveness of the multimodal NCL against existing multimedia and multimodal languages, for both input and output modalities.

Guest Editorial Special Section on Visual Saliency Computing and Learning

Vision and multimedia communities have long attempted to enable computers to understand image or video content in a manner analogous to humans. Humans’ comprehension to an image or a video clip often depends on the objects that draw their attention. As a result, one fundamental and open problem is to automatically infer the attention attracting or interesting areas in an image or a video sequence. Recently, a large number of researchers explore visual saliency models to address this problem. The study on visual saliency models is originally motivated by simulating humans’ bottom-up visual attention and it is mainly based on the biological evidence that humans’ visual attention is automatically attracted by highly salient features in the visual scene, which are discriminative with respect to the surrounding environment.

Interview with Dick Bulterman

Dick Bulterman is a professor of computer science at the Vrije Universiteit Amsterdam and a senior researcher at CWI: Centrum Wiskunde en Informatica, also in Amsterdam, where he has worked since 1988. From 2013--2015 he was President and CEO of FX Palo Alto Laboratories in California. From 1988--2013, he was head of the group Distributed and Interactive Systems. Between 1997 and 2002, Bulterman was the Managing Director of Oratrix Development bv, a CWI spin-off technology company. Before joining CWI, he was a professor of computer engineering at Brown University in Providence, RI. Bulterman has been active in the interactive television systems area for many years and has done work together with his CWI colleagues on a wide range of user interface topics (including early work on the use of secondary screens for personalizing media content). Bulterman has been active in the multimedia community since 1993 and has served in various roles on ACM MM organizing committees. He was Vice-Chair of ACM SIGWEB before becoming Chair in 2015. He was a founding editorial board member of ACM TOMCCAP and ACM/Springer Multimedia Tools and Applications, and is associate editor of Springer Multimedia Systems. He is past chair of W3C's Synchronized Multimedia working group. Bulterman received a Ph.D. in computer science from Brown University in 1981.