Multimedia Community Research Articles

Introduction to the Special Section on Contextual Object Analysis in Complex Scenes

In Recent years, with the vast development of deep learning techniques, a great deal of effort has been devoted in the computer vision and multimedia community toward the problems of visual object analysis, such as object representation, recognition, detection, identification, etc. Especially at the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) in 2014, the computers have successfully outperformed humans with a lower error rate at image recognition for the first time. However, most of existing algorithms focus more on analyzing objects in a relatively simple and restricted situation, which may perform poorly in natural environments.

Hybrid-Attention Enhanced Two-Stream Fusion Network for Video Venue Prediction

Video venue category prediction has been drawing more attention in the multimedia community for various applications such as personalized location recommendation and video verification. Most of existing works resort to the information from either multiple modalities or other platforms for strengthening video representations. However, noisy acoustic information, sparse textual descriptions and incompatible cross-platform data could limit the performance gain and reduce the universality of the model. Therefore, we focus on discriminative visual feature extraction from videos by introducing a hybrid-attention structure. Particularly, we propose a novel Global-Local Attention Module (GLAM), which can be inserted to neural networks to generate enhanced visual features from video content. In GLAM, the Global Attention (GA) is used to catch contextual scene-oriented information via assigning channels with various weights while the Local Attention (LA) is employed to learn salient object-oriented features via allocating different weights for spatial regions. Moreover, GLAM can be extended to ones with multiple GAs and LAs for further visual enhancement. These two types of features respectively captured by GAs and LAs are integrated via convolution layers, and then delivered into convolutional Long Short-Term Memory (convLSTM) to generate spatial-temporal representations, constituting the content stream. In addition, video motions are explored to learn long-term movement variations, which also contributes to video venue prediction. The content and motion stream constitute our proposed Hybrid-Attention Enhanced Two-Stream Fusion Network (HA-TSFN). HA-TSFN finally merges the features from two streams for comprehensive representations. Extensive experiments demonstrate that our method achieves the state-of-the-art performance in the large-scale dataset Vine. The visualization also shows that the proposed GLAM can capture complementary scene-oriented and object-oriented visual features from videos. Our code is available at: https://github.com/zhangyanchao1014/HA-TSFN .

Report from the MMM 2020 Special Session on Multimedia Datasets for Repeatable Experimentation (MDRE 2020)

Information retrieval and multimedia content access have a long history of comparative evaluation, and many of the advances in the area over the past decade can be attributed to the availability of open datasets that support comparative and repeatable experimentation. Hence, sharing data and code to allow other researchers to replicate research results is needed in the multimedia modeling field, as it helps to improve the performance of systems and the reproducibility of published papers. This report summarizes the special session on Multimedia Datasets for Repeatable Experimentation (MDRE 2020), which was organized at the 26th International Conference on MultiMedia Modeling (MMM 2020), held in January 2020 in Daejeon, South Korea. The intent of these special sessions is to be a venue for releasing datasets to the multimedia community and discussing dataset related issues. The presentation mode in 2020 was to have short presentations (approximately 8 minutes), followed by a panel discussion moderated by Aaron Duane. In the following we summarize the special session, including its talks, questions, and discussions.

Guest Editorial Multimedia Computing With Interpretable Machine Learning

The papers in this special section is to broadly engage the machine learning and multimedia communities on the emerging yet challenging interpretable machine learning. Multimedia is increasingly becoming the “biggest big data,” among the most important and valuable source for insight and information. Many powerful machine learning algorithms, especially deep learning models such as convolutional neural networks (CNNs), have recently achieved outstanding predictive performance in a wide range of multimedia applications, including visual object classification, scene understanding, speech recognition, and activity prediction. Nevertheless, most deep learning algorithms are generally conceived as blackbox methods, and it is difficult to intuitively and quantitatively understand the results of their prediction and inference. Since this lack of interpretability is a major bottleneck in designing more successful predictive models and exploring wider-range useful applications, there has been an explosion of interest in interpreting the representations learned by these models, with profound implications for research into interpretable machine learning in the multimedia community.

Fashion analysis and understanding with artificial intelligence

As handling fashion big data with Artificial Intelligence (AI) has become exciting challenges for computer scientists, fashion studies have received increasing attention in computer vision, machine learning and multimedia communities in the past few years. In this paper, introduce the progress in fashion research and provide a taxonomy of these fashion studies that include low-level fashion recognition, middle-level fashion understanding and high-level fashion applications. Finally, we discuss the challenges that when the fashion industry faces AI technologies.

Multiscale summarization and action ranking in egocentric videos

Useful information extraction from egocentric videos has evolved as an important research problem for both computer vision and multimedia communities. In this paper, we have addressed two problems, namely (i) generating multiscale summaries, i.e., multiple summaries of different lengths and (ii) priority-based ranking of various actions present in egocentric videos. A new algorithm, termed as Multiscale Egocentric Video Summarization and Action Ranking (MEVSAR), with agglomerative clustering as its backbone, is proposed to solve the above problems. Importantly, the MEVSAR algorithm follows an “analyze once, generate many” principle to generate multiple summaries in a single run and subsequently rank actions from the generated summaries. Experimental evaluation on two well-known publicly available datasets clearly demonstrate the merits of the proposed approach.

Exploring Global and Local Linguistic Representations for Text-to-Image Synthesis

The task of text-to-image synthesis is to generate photographic images conditioned on given textual descriptions. This challenging task has recently attracted considerable attention from the multimedia community due to its potential applications. Most of the up-to-date approaches are built based on generative adversarial network (GAN) models, and they synthesize images conditioned on the global linguistic representation. However, the sparsity of the global representation results in training difficulties on GANs and a shortage of fine-grained information in the generated images. To address this problem, we propose cross-modal global and local linguistic representations-based generative adversarial networks (CGL-GAN) by incorporating the local linguistic representation into the GAN. In our CGL-GAN, we construct a generator to synthesize the target images and a discriminator to judge whether the generated images conform with the text description. In the discriminator, we construct the cross-modal correlation by projecting the image representations at high and low levels onto the global and local linguistic representations, respectively. We design the hinge loss function to train our CGL-GAN model. We evaluate the proposed CGL-GAN on two publicly available datasets, the CUB and the MS-COCO. The extensive experiments demonstrate that incorporating fine-grained local linguistic information with cross-modal correlation can greatly improve the performance of text-to-image synthesis, even when generating high-resolution images.

Singaporean Societies: Multimedia Communities of Student Migration

As young Singaporeans are evaluating their obligations towards their parents at home, the state of Singapore is implementing policies to entrench long-term connection between overseas Singaporean students and their families by using nancial support to guide over- seas Singaporean student societies. These methods reach far beyond Singapore’s borders and involve a combination of online and of ine communities of practice that bring young overseas Singaporeans closer together by setting social boundaries across multiple media. Young Singaporeans learn about studying overseas through online communities, and Sin- gaporean societies seek to control that form of communication. In this paper, the author describes the worldwide state-funded and student-run Singaporean societies and how they seek to govern overseas students’ relationships with family at home using methods such as social media, nances, and parties. Drawing from ethnographic and online methods of inquiry over three months in 2015, this article explores how students experienced Singa- porean societies as a tool to access social and nancial resources, which set boundaries for them when reevaluating their responsibilities at home while they live abroad. The author looks at the critical language that is present in an online community of young Singaporeans and shows how Singaporean societies limit opportunities for criticism.

Singaporean Societies: Multimedia Communities of Student Migration

Studies of home language use tend to focus on macro-level language changes and verbal communication rather than on micro-level analyses of family communication. This makes it diffcult to form a nuanced picture of the relations among diverse resources deployed in transnational family communication. In this paper, we address this issue by reporting results from a preliminary study of the micro-level communicative interactions of a frst-generation transnational Australian Vietnamese family who have settled in Melbourne, Australia. Through an in-depth analysis of a 20-minute video clip, we capture intersections in the rich, diverse communicative resources used by the family as they watched a favourite English television program while simultaneously keeping in contact with family members overseas. Our analysis shows that transnational family communication patterns involve complex displays of language use, silence, touch, movement, and spatial orientation, which together enable the family to communicate in the here and now with individuals near and far. We use the multiplicity framework to interpret the fuidity of multimodal communication, intimacy, and continuity across space.

Opinion Column: Evolution of topics in the multimedia community

For this edition of the SIGMM Opinion Column, we asked members of the Multimedia community to share their impressions about the shift of scientific topics in the community over the years, namely the evolution of "traditional" and "emerging" Multimedia topics. This subject has emerged in several conversations over the 2 years of history of the SIGMM Opinion Column, and we report here a summary of recent and old discussions, happened over different channels --- our Facebook group, the SIGMM Linkedin group, and in-person conversations between the column editors and MM researchers --- with hopes, fears and opinions around this problem. We want to thank all participants for their precious contribution to these discussions.

S3D: Scalable Pedestrian Detection via Score Scale Surface Discrimination

Pedestrian detection has remained an important research topic in both the computer vision and multimedia communities because of its importance in practical applications, such as driving assistance and video surveillance. Existing methods compare the response score with a fixed threshold to determine whether a candidate region contains pedestrians and produce dissatisfactory results that contain either missed detections or false detections, which are difficult to balance. This situation has a serious impact under the condition of variable scale. This paper investigates the functional relationship between the scores and scales of pedestrians. By designing experiments with multiple scales, we have found a discriminant surface in the score scale space. Pedestrians can be distinguished at various scale levels according to their locations on the discriminant surface. The proposed approach is evaluated using four challenging pedestrian detection datasets, including Caltech, INRIA, ETH, and KITTI, and the superior experimental results are achieved when compared with baseline methods.

Deep Memory Network for Cross-Modal Retrieval

With the explosive growth of multimedia data on the Internet, cross-modal retrieval has attracted a great deal of attention in both computer vision and multimedia communities. However, this task is challenging due to the heterogeneity gap between different modalities. Current approaches typically involve a common representation learning process that maps data from different modalities into a common space by linear or nonlinear embedding. Yet, most of them only handle the dual-modal situation and generalize poorly to complex cases that involve multiple modalities. In addition, they often require expensive fine-grained alignment of training data among diverse modalities. In this paper, we address these with a novel cross-modal memory network (CMMN), in which memory contents across modalities are simultaneously learned from end to end without the need of exact alignment. We further account for the diversity across multiple modalities using the strategy of adversarial learning. Extensive experimental results on several large-scale datasets demonstrate that the proposed CMMN approach achieves state-of-the-art performance in the task of cross-modal retrieval.

Deep Learning–Based Multimedia Analytics

The multimedia community has witnessed the rise of deep learning–based techniques in analyzing multimedia content more effectively. In the past decade, the convergence of deep-learning and multimedia analytics has boosted the performance of several traditional tasks, such as classification, detection, and regression, and has also fundamentally changed the landscape of several relatively new areas, such as semantic segmentation, captioning, and content generation. This article aims to review the development path of major tasks in multimedia analytics and take a look into future directions. We start by summarizing the fundamental deep techniques related to multimedia analytics, especially in the visual domain, and then review representative high-level tasks powered by recent advances. Moreover, the performance review of popular benchmarks gives a pathway to technology advancement and helps identify both milestone works and future directions.

Ancillary Benefit of Increased HPV Immunization Rates Following a CBPR Approach to Address Immunization Disparities in Younger Siblings.

Increasing HPV vaccination rates may decrease the disproportionately high HPV-associated disease incidence and mortality in African Americans (AA) and lower socioeconomic individuals. Data from a community-based participatory research (CBPR) study addressing immunization disparities among 19-35month old children was analyzed to identify ancillary benefits in HPV immunization rates for adolescent siblings. Sub-study analysis inclusion criteria: AA (N = 118), 13-17years old, younger sibling enrolled in parent study, and enrolled ≥ 9months. Parent/caregiver interventions included: a web-based immunization toolkit with information on age-appropriate vaccines; a multimedia community outreach campaign; and reminder mailings. HPV up-to-date (UTD) status was defined as Wisconsin Immunization Registry (WIR) documentation of at least three HPV vaccines. McNemar's test compared pre/post intervention HPV status. Two dependent proportions testing compared the proportion of adolescents that became UTD in the study cohort, City of Milwaukee, and State of Wisconsin. Parents/caregivers perceived that 92% of adolescents were HPV-UTD, while only 24% had a WIR-verified HPV-UTD status. Baseline UTD status of the younger siblings 19-35month old 4:3:1:3:3:1:4 antigen series was 63%, which increased to 86% at study completion. Adolescent's HPV-UTD immunization status increased from 30 (25%) at enrollment to 54 (46%) at study completion [p = 0.004]. A statistically significant larger proportion of adolescents became HPV-UTD in the study cohort (20%) compared to the City of Milwaukee [14%, p = 0.042] and the State of Wisconsin [14%, p = 0.046]. A culturally-tailored CBPR approach targeting parents/caregivers of younger AA children can have significant ancillary benefit to increase HPV immunization rates in adolescent siblings.

Affective image classification by jointly using interpretable art features and semantic annotations

Affective image classification, which aims to classify images according to their affective characteristics of inducing human emotions, has drawn increasing research attentions in the multimedia community. Although many features have been attempted, the semantic gap between low-level visual features and high-level emotional semantics, however, remains a major challenge. In this paper, we propose an affective image classification algorithm by jointly using the visual features extracted under the guidance of the art theory and semantic image annotations, such as the categories of objects and scenes, generated by a pre-trained deep convolutional neural network. This algorithm has been evaluated against three state-of-the-art approaches on three benchmark image datasets. Our results indicate that combining interpretable aesthetic features and semantic annotations can better characterize the emotional semantics and the proposed algorithm is able to produce more accurate affective image classification than the other three approaches.

Towards data driven conferences within SIGMM

There is no doubt that research in our field has become more data driven. And while the term data science might suggest there is some science without data, it is data the feeds our systems, trains our networks, and tests our results. Recently we have seen examples of a few conferences experimenting with their review process to gain new insights. For those of us in the Multimedia (MM) community, it is not an entirely new thing. In 2013, I led an effort along with my TPC Co-Chairs to look at the past several years of conferences and examine the reviewing system, the process, the scores, and the reviewer load. Additionally, we ran surveys to the authors (accepted and rejected) and the reviewers and ACs to gauge how we did. This was presented at the business meeting in Barcelona along with the suggestion that this practice continues. While this was met with great praise, it was never repeated.

SIGMM records

A new editorial team has committed to lead the ACM SIGMM Records since the issue of January 2017. The goal is to consolidate the Records as a primary source of information and a communication vehicle for the multimedia community. With these objectives in mind, the Records were re-organized around three main categories (Open Science, Information, and Opinion), for which specific sections and columns were created (more details in http://records.mlab.no/2017/05/08/sigmm-records-serving-the-community/).

A deep generic to specific recognition model for group membership analysis using non-verbal cues

Automatic understanding and analysis of groups has attracted increasing attention in the vision and multimedia communities in recent years. However, little attention has been paid to the automatic analysis of the non-verbal behaviors and how this can be utilized for analysis of group membership, i.e., recognizing which group each individual is part of. This paper presents a novel Support Vector Machine (SVM) based Deep Specific Recognition Model (DeepSRM) that is learned based on a generic recognition model. The generic recognition model refers to the model trained with data across different conditions, i.e., when people are watching movies of different types. Although the generic recognition model can provide a baseline for the recognition model trained for each specific condition, the different behaviors people exhibit in different conditions limit the recognition performance of the generic model. Therefore, the specific recognition model is proposed for each condition separately and built on top of the generic recognition model. A number of experiments are conducted using a database aiming to study group analysis while each group (i.e., four participants together) were watching a number of long movie segments. Our experimental results show that the proposed deep specific recognition model (44%) outperforms the generic recognition model (26%). The recognition of group membership also indicates that the non-verbal behaviors of individuals within a group share commonalities.

Weakly Supervised Multimodal Hashing for Scalable Social Image Retrieval

Recent years have witnessed a dramatic increase in the number of community-contributed images. Hashing-based similarity searches for social images have been attracting considerable interest from computer vision and multimedia communities due to their computational and memory efficiency. In this paper, we propose a novel weakly supervised hashing method named weakly supervised multimodal hashing, for scalable social image retrieval. Semantic-aware hash functions are learned by jointly leveraging the weakly supervised tag information and visual information. Specifically, because user-provided tags associated with social images can describe the semantic information, the hash functions are learned by exploring the semantic structure. Unfortunately, the user-provided tags are imperfect. To avoid overfitting the weakly supervised tags, the local discriminative structure and the geometric structure in the visual space are explored. Besides, to learn compact and non-redundant hash codes, the hash functions are constrained to be orthogonal and an information theoretic regularization based on the maximum entropy principle is introduced to maximize the information provided by each hash code. The learned hash functions are orthogonal, which can avoid redundancy in the learned hash codes as much as possible. The proposed hashing learning problem is formulated as the eigenvalue problem, which can be solved efficiently. Extensive experiments are conducted on two widely used social image data sets and the encouraging performance compared with the state-of-the-art hashing techniques demonstrates the effectiveness of the proposed method.

Deep learning with particle filter for person re-identification

Person re-identification, having attracted much attention in the multimedia community, is still challenged by the accuracy and the robustness, as the images for the verification contain such variations as light, pose, noise and ambiguity etc. Such practical challenges require relatively robust and accurate feature learning technologies. We introduced a novel deep neural network with PF-BP(Particle Filter-Back Propagation) to achieve relatively global and robust performances of person re-identification. The local optima in the deep networks themselves are still the main difficulty in the learning, in despite of several advanced approaches. A novel neural network learning, or PF-BP, was first proposed to solve the local optima problem in the non-convex objective function of the deep networks. When considering final deep network to learn using BP, the overall neural network with the particle filter will behave as the PF-BP neural network. Also, a max-min value searching was proposed by considering two assumptions about shapes of the non-convex objective function to learn on. Finally, a salience learning based on the deep neural network with PF-BP was proposed to achieve an advanced person re-identification. We test our neural network learning with particle filter aimed to the non-convex optimization problem, and then evaluate the performances of the proposed system in a person re-identification scenario. Experimental results demonstrate that the corresponding performances of the proposed deep network have promising discriminative capability in comparison with other ones.