Multimedia Research Research Articles

Video Storytelling: Textual Summaries for Events

Bridging vision and natural language is a longstanding goal in computer vision and multimedia research. While earlier works focus on generating a single-sentence description for visual content, recent works have studied paragraph generation. In this work, we introduce the problem of video storytelling, which aims at generating coherent and succinct stories for long videos. Video storytelling introduces new challenges, mainly due to the diversity of the story and the length and complexity of the video. We propose novel methods to address the challenges. First, we propose a context-aware framework for multimodal embedding learning, where we design a Residual Bidirectional Recurrent Neural Network to leverage contextual information from past and future. Second, we propose a Narrator model to discover the underlying storyline. The Narrator is formulated as a reinforcement learning agent which is trained by directly optimizing the textual metric of the generated story. We evaluate our method on the Video Story dataset, a new dataset that we have collected to enable the study. We compare our method with multiple state-of-the-art baselines, and show that our method achieves better performance, in terms of quantitative measures and user study.

Characterization and classification of semantic image-text relations

The beneficial, complementary nature of visual and textual information to convey information is widely known, for example, in entertainment, news, advertisements, science, or education. While the complex interplay of image and text to form semantic meaning has been thoroughly studied in linguistics and communication sciences for several decades, computer vision and multimedia research remained on the surface of the problem more or less. An exception is previous work that introduced the two metrics Cross-Modal Mutual Information and Semantic Correlation in order to model complex image-text relations. In this paper, we motivate the necessity of an additional metric called Status in order to cover complex image-text relations more completely. This set of metrics enables us to derive a novel categorization of eight semantic image-text classes based on three dimensions. In addition, we demonstrate how to automatically gather and augment a dataset for these classes from the Web. Further, we present a deep learning system to automatically predict either of the three metrics, as well as a system to directly predict the eight image-text classes. Experimental results show the feasibility of the approach, whereby the predict-all approach outperforms the cascaded approach of the metric classifiers.

I Speak for the Water: Anishinaabe Nibi Grandmothers and Watershed Discipleship in Northern Midwestern Protestant Communities

I Speak for the Water: Anishinaabe Nibi Grandmothers and Watershed Discipleship in Northern Midwestern Protestant Communities Kiara Jorgenson (bio) Ne-be Gee Zah- gay- e- gooGee Me-gwetch -wayn ne- me—gooGee Zah Wayn ne- me- goo1 Every Sunday morning Sharon Day gathers near the waters of the Gichi Ziibi (Mississippi River) at Hidden Falls Regional Park in St. Paul, MN, to sing prayers for the river. Much like the words of her sister’s prayer above, Sharon’s prayers offer love, gratitude, and respect to the waters. Though her shared company changes from week to week, she is rarely alone in her efforts as she has extended an invitation for any and all to join in this spiritual ritual of reverence and reciprocity. Sharon’s longstanding leadership with the Indigenous Peoples Taskforce and role as a second degree M’dewiwin of the Bois Forte Band of Ojibwe in northern Minnesota have positioned her as a powerful elder in northern Midwestern indigenous communities and beyond, but it has been her particular work related to the protection and preservation of waters that has distinguished her as one of few celebrated Nibi (water) grandmothers. THE IMPORTANCE OF INTENTION SETTING & RITUAL PROTOCOLS In settings like those Sunday mornings beside the Gichi Ziibi and on formal Nibi Walks, Ojibwe elders like Sharon generously share traditional ecological knowledge within and without their communities, often leading educators and activists, clergy and lobbyists to a deeper understanding of water’s dignity and the collective call to care for its future (see figure 1). As a type of spiritual-ecological resistance movement (my term, not Sharon’s) Nibi Walks are particularly powerful given recent infringements upon the water rights of many indigenous groups, including the Ojibwe of Fond du Lac Reservation in northern Minnesota where Enbridge, a multinational energy transportation [End Page 194] Click for larger view View full resolution Fig. 1. Sharon Day, Executive Director of the Indigenous People’s Taskforce and curator of the Nibi Walks website. Courtesy of Sharon Day, nibiwalk.org. company, seeks to complete construction of the controversial Line 3 crude oil pipeline. But perhaps equally valuable to the lessons learned on Nibi Walks is the very way Nibi grandmothers like Sharon gather to honor lakes and rivers, oceans and watersheds. As I shall seek to illustrate below, their model of water walking showcases the spiritual value of prayerful presence and intentional connection with one’s place. With water, as with all facets of a responsible life, the intentions and rituals that make them plain prove paramount. A good example of how method can itself be a spiritual lesson is the Nibi Walk orientation, a half-day of guidance wherein the spiritual goal of Nibi Walks is clearly explained and related protocols outlined. As inherently inclusive Nibi Walks are not M’dewiwin spiritual ceremonies or religious practices in a qualified sense, but neither are they political events or environmental rallies devoid of spiritual tenor. In Sharon’s words, “Nibi Walks are about prayer, not protest.” And because they connect people from divergent contexts and places, the walks require protocols to clarify purpose. During a Nibi Walk water is carried in a copper pail, to remind all involved of Earth’s elements in its tender beginnings. Water, itself rendered an inanimate noun in Ojibwe language as opposed to animate rocks, trees and plants, is to be carried in relay, moving continuously like the river itself [End Page 195] and always in close proximity to the snaking, forging current. As privileged and anointed carriers of the same primordial water that birthed the cosmos, women carry the enclosed water wearing skirts to demonstrate both ceremonial intent and respect for Shkaakaamikwe (Mother Earth), understood as inseparable from one’s very self. If men are present on a walk they carry the eagle feather staff, a symbol of protection for the journey. Finally, endangered Ojibwe language is used whenever possible throughout the walk, its spoken form a prayerful consecration. When I presented the multi-media research which serves as the foundation of this essay in San Diego at the 2019 American Academy of Religion’s “Christian Spirituality Unit,” I also began by setting an...

Food Recommendation: Framework, Existing Solutions, and Challenges

A growing proportion of the global population is becoming overweight or obese, leading to various diseases (e.g., diabetes, ischemic heart disease and even cancer) due to unhealthy eating patterns, such as increased intake of food with high energy and high fat. Food recommendation is of paramount importance to alleviate this problem. Unfortunately, modern multimedia research has enhanced the performance and experience of multimedia recommendation in many fields such as movies and POI, yet largely lags in the food domain. This article proposes a unified framework for food recommendation, and identifies main issues affecting food recommendation including building the personal model, analyzing unique food characteristics, incorporating various context and domain knowledge. We then review existing solutions for these issues, and finally elaborate research challenges and future directions in this field. To our knowledge, this is the first survey that targets the study of food recommendation in the multimedia field and offers a collection of research studies and technologies to benefit researchers in this field.

Movienet: a movie multilayer network model using visual and textual semantic cues

Discovering content and stories in movies is one of the most important concepts in multimedia content research studies. Network models have proven to be an efficient choice for this purpose. When an audience watches a movie, they usually compare the characters and the relationships between them. For this reason, most of the modelsdeveloped so far are based on social networks analysis. They focus essentially on the characters at play. By analyzing characters interactions, we can obtain a broad picture of the narration’s content. Other works have proposed to exploit semantic elements such as scenes, dialogues, etc.. However, they are always captured from a single facet. Motivated by these limitations, we introduce in this work a multilayer network model to capture the narration of a movie based on its script, its subtitles, and the movie content. After introducing the model and the extraction process from the raw data, weperform a comparative analysis of the whole 6-movie cycle of the Star Wars saga. Results demonstrate the effectiveness of the proposed framework for video content representation and analysis.

Can the multimedia research community via quality of experience contribute to a better quality of life?

Can the multimedia research community via quality of experience contribute to a better quality of life?

Integration Borg & Gall (1983) and Lee & Owen (2004) models as an alternative model of design-based research of interactive multimedia in elementary school

Interactive multimedia has different development characteristics with other media. This article aims to form an interactive multimedia development model in elementary schools by integrating the development model of Borg & Gall and Lee & Owen. The formulation of these steps is the integration of the development model of Borg & Gall (1983) and Lee & Owen (2004). Borg & Gall focuses on developing new research-based, ready-to-use research products and Lee & Owen that are more specific in developing interactive multimedia products. Through the analysis of the advantages and limitations of each stage of Borg & Gall and Lee & Owen models can be formulated integration steps become as follows: (1) Research and Information Collecting; (2) Planning; (3) Development of Preliminary Form of Product; (4) Expert Evaluation; (5) Revisions Based on Expert Evaluation Results (Revision 1); (6) Preliminary Field Testing; (7) Main Product Revision (Revision 2); (8) Main Field Testing; (9) Operational Product Revision (Revision 3); (10) Operational Field Testing; (11) Final Product Revision (Revision 4); (12) Dissemination and Implementation. The considerations of the researchers to formulate the 12 stages are: (1) advantages of the comprehensive Borg & gall model, with a research-based product development stage and very rigorous testing phases, (2) the advantages of Lee & Owen’s model that focuses on development principles interactive multimedia, (3) the trend of interactive multimedia development research in the scope of elementary schools, and (4) the development of the era that increasingly demands technology-based media for students. The twelve stages formulated by the author above can be adjusted or modified by reducing/adding/changing according to the need to maximize research activities by other.

Multimedia for Autonomous Driving

Multimedia has played an indispensable role in the success of various real-world applications, from smart healthcare to intelligent surveillance systems. In this new era of technology, one of these essential and useful applications is autonomous driving or self-driving cars. The idea of driverless vehicles was introduced many years ago when autopilot systems were designed for airplanes. However, over the past few years, there have been incredible advances in autonomous driving, thanks to new technologies in multimedia and artificial intelligence. In particular, multimedia research using a broad range of techniques such as machine learning, computer vision, audio processing, simulation systems, robotics, and big data has significantly enhanced and expedited the autonomous driving technology over time. In addition to the main applications of self-driving cars such as autonomous delivery vehicles, taxis, and freight trucks, the autonomous driving technology can provide many opportunities in military, constructions, healthcare, and emergency management. Despite the effectiveness and efficiency that self-driving cars bring to the future of transportation, many challenges remain that need to be solved before fully-autonomous driving becomes a reality.

Fine-Grained Land Use Classification at the City Scale Using Ground-Level Images

Multimedia researchers have exploited large collections of community-contributed geo-referenced images to better understand a particular image, such as its subject matter or where it was taken, as well as to better understand a geographic location, such as the most visited tourist spots in a city or what the local cuisine is like. The goal of this paper is to better understand location. In particular, we use geo-referenced image collections to better understand what occurs in different parts of a city at fine spatial and activity class scales. This problem is known as land use mapping in the geographical sciences. We propose a novel framework to perform fine-grained land use mapping at the city scale using ground-level images. Mapping land use is considerably more difficult than mapping land cover and is generally not possible using overhead imagery as it requires close-up views and seeing inside buildings. We postulate that the growing collections of geo-referenced, ground-level images suggest an alternate approach to this geographic knowledge discovery problem. We develop a general framework that uses Flickr images to map 45 different land-use classes for the city of San Francisco, CA, USA. Individual images are classified using a novel convolutional neural network containing two streams: one for recognizing objects and another for recognizing scenes. This network is trained in an end-to-end manner directly on the labeled training images. We propose several novel strategies to overcome the noisiness of our user-generated data including search-based training set augmentation and online adaptive training. We derive a ground truth map of San Francisco in order to evaluate our method. We demonstrate the effectiveness of our approach through geovisualization and quantitative analysis. Our framework achieves over 29% recall at the individual land parcel level that represents a strong baseline for the challenging 45-way land use classification problem, especially given the noisiness of the image data.

Text extraction from natural scene images using Renyi entropy

Nowadays, development in machine vision incorporated with artificial intelligence surpasses the ability of human intelligence and its application expands exponentially with the increasing number of electronic gadgets in our day-to-day life. The explosive revolution in multimedia research leads to the need for expanding the utility of texts in a machine vision environment to promote web search operation. Hence, extracting text from images forms the core aspect of information retrieval-based intelligent system. This article is aimed towards extracting text from unconstrained environments. Here, the significance of the CIE-Lab colour space is analysed over text localisation assisted through Renyi entropy-based thresholding. The proposed algorithm is tested on the MSRA Text Detection 500 dataset (MSRA-TD500) and Street View Text (SVT) datasets, which are challenging datasets. Authors’ proposed Renyi entropy-based text localisation algorithm is successful in identifying blurred texts, texts with different font characteristics and multi-lingual texts with manifold orientations from complex background natural scenes.

Is Artificial Intelligence New to Multimedia?

The field of multimedia covers a broad range of research and technologies that aim to develop solutions for individual disciplines as well as multidisciplinary domains. From the beginning, multimedia research has employed Artificial Intelligence (AI) techniques to address various challenges in this area. Specifically, AI has been extensively used for multimedia retrieval, management, and analysis. Over the past few years, multimedia has continued to utilize AI to bring revolutionary advancement in numerous applications and domains, mainly thanks to the recent progress in machine learning algorithms and computing powers. However, many challenges remain, which calls for the need to extend Multimedia AI to tackle these problems and to lead to great opportunities in the near future.

Pushing the Boundary of Multimedia Big Data: An Overview of IEEE MIPR

New forms of multimedia data (such as text, numbers, tags, networking, signals, geo-tagged information, graphs/relationships, three-dimensional/VR/AR and sensor data, etc.) have emerged in many applications in addition to traditional multimedia data (image, video, audio). Multimedia has become the biggest of big data as the foundation of today's data-driven discoveries. Almost all disciplines of science and engineering, as well as social sciences, involve multimedia data in some forms, such as recording experiments, driverless cars, unmanned aerial vehicles, smart communities, biomedical instruments, security surveillance. Some recent events demonstrate the power of real-time broadcast of unfolding events on social networks. Multimedia data is not just big in volume, but also multi-modal and mostly unstructured. Storing, indexing, searching, integrating, and recognizing from the vast amounts of data create unprecedented challenges. Even though significant progress has been made processing multimedia data, today's solutions are inadequate in handling data from millions of sources simultaneously. The IEEE International Conference on Multimedia Information Processing and Retrieval (IEEE-MIPR) aims to provide a forum for original research contributions and practical system design, implementation, and applications of multimedia information processing and retrieval for single modality or multiple modalities. The target audiences are university researchers, scientists, industry practitioners, software engineers, and graduate students who want to become acquainted with technologies for big data analytics, machine intelligence, information fusion in multimedia information processing and retrieval. Following a successful debut in Miami, Florida in 2018, the second MIPR, held in San Jose, California, attracted over 140 submissions and more than 100 attendees—a big growth from its inaugural event and an indication of its growing visibility and popularity in the multimedia research society. In addition to four keynote talks, MIPR'19 also featured six innovation multimedia forums organized by both academic and industrial researchers, with topics ranging from computer vision, self-driving, Natural Language Processing, next generation video display, to cutting edge video/audio coding techniques.

ACM Multimedia 2019 and Reproducibility in Multimedia Research

The first months of the new calendar year, multimedia researchers traditionally are hard at work on their ACM Multimedia submissions. (This year the submission deadline is 1 April.) Questions of reproducibility, including those of data set availability and release, are at the forefront of everyone's mind. In this edition of SIGMM Records, the editors of the "Data Sets and Benchmarks" column have teamed up with two intersecting groups, the Reproducibility Chairs and the General Chairs of ACM Multimedia 2019, to bring you a column about reproducibility in multimedia research and the connection between reproducible research and publicly available data sets. The column highlights the activities of SIGMM towards implementing ACM paper badging. ACM MMSys has pushed our community forward on reproducibility and pioneered the use of ACM badging [1]. We are proud that in 2019 the newly established Reproducibility track will introduce badging at ACM Multimedia. Complete information on Reproducibility at ACM Multimedia is available at: https://project.inria.fr/acmmmreproducibility/

Processing textual and visual certainty information about digital architectural models

Based on multimedia research and research about hedging and visualization styles, we examined the interplay between the certainty of text and picture information on the perceived certainty of digital architectural models and knowledge acquisition. The information to be learned was taken from an archaeological scientific article. Learners were able to link certainty information better from the text to the corresponding elements in the detailed picture than to the corresponding elements in the line drawing. Text information presented with formulations of certainty was better retained than text information presented with formulations of uncertainty. This indicates that text information presented with certainty formulations is valued more with regard to the processing goal and therefore better attended to and learned than text information presented with uncertainty formulations. In sum, although certainty formulations had a greater influence on processing than visualizations, there were different influences of visualization style on knowledge acquisition. Here the determinant was the content to be processed: certainty information, visualized content information, or non-visualized content information.

Construction of Diverse Image Datasets From Web Collections With Limited Labeling

Image datasets play a pivotal role in advancing computer vision and multimedia research. However, most of the datasets are created by extensive human effort and are extremely expensive to scale-up. To address these issues, several automatic and semi-automatic approaches have been proposed for creating datasets by refining web images. However, these approaches either include significant redundant images in the dataset or fail to provide a diverse enough set to train a robust classifier. Ideally, a representative subset should be both semantically and visually diverse so as to provide the maximum amount of information under the current budget. Most current approaches are entirely based on the analysis of visual features, which may not correlate well with image semantics, and hence, collected images may not be sufficient to give a detailed understanding of a category. In this paper, we propose a system for creating diverse image dataset collections from the web with limited manual labeling effort. It is based upon a semi-supervised sparse coding framework that employs a joint visual-semantic space to simultaneously utilize both the images and associated textual information from the web for dataset construction. In addition, the proposed system is online and is capable of collecting more discriminative images continuously as new data becomes available, which is also suitable for enriching the existing datasets. The experiments demonstrate that our system can create and enrich datasets with limited manual labeling, with better cross-dataset generalization capability and diversity compared to the state-of-the-art datasets.

New Modality: Emoji Challenges in Prediction, Anticipation, and Retrieval

Over the past decade, emoji have emerged as a new and widespread form of digital communication, spanning diverse social networks and spoken languages. We propose treating these ideograms as a new modality in their own right, distinct in their semantic structure from both the text in which they are often embedded as well as the images which they resemble. As a new modality, emoji present rich novel possibilities for representation and interaction. In this paper, we explore the challenges that arise naturally from considering the emoji modality through the lens of multimedia research, specifically the ways in which emoji can be related to other common modalities such as text and images. To do so, we first present a large-scale data set of real-world emoji usage collected from Twitter. This data set contains examples of both text-emoji and image-emoji relationships within tweets. We present baseline results on the challenge of predicting emoji from both text and images, using state-of-the-art neural networks. Further, we offer a first consideration into the problem of how to account for new, unseen emoji—a relevant issue as the emoji vocabulary continues to expand on a yearly basis. Finally, we present results for multimedia retrieval using emoji as queries.

Developing multimedia collaboratively: Practical approaches for large-scale online curriculum development

Transformations in contemporary higher education have led to an explosion in the number of degrees delivered online, a significant characteristic of which is the incorporation of multimedia to support learning. Despite the proliferation of multimedia and growing literature about the affordances of various technologies, there are relatively few examples of how judgements are made regarding choosing and actioning multimedia development decisions for educational developers. The case study presented here is framed within an institution-wide project for the development of fully online degrees that utilised a collaborative approach to curriculum and multimedia development. This example focuses on the establishment and operation of a collaborative approach to curriculum development in which multidisciplinary development teams invested considerable resources in researching improvements to their multimedia practices and processes. This article reflects on the collaborative team approach to multimedia design and development by examining the team’s experiences and practices through the lens of existing multimedia research, in order to understand the convergence between multimedia theory and the practicalities of developing multimedia within the constraints of large-scale online curriculum development. Through these reflections, four lessons learned will be explicated which will inform those engaged in employing similar approaches in other contexts. These lessons learned identify the benefits and potential issues associated with: the approach used by the collaborative development team to support the production of multimedia, the practices and process used by the collaborative development team to facilitate the creation of concise multimedia presentations, the impacts of establishing teaching presence through videos created by the course writer and online course facilitator, and the presentation styles used by course writers and the tools they used during multimedia production.

Implementing Restorative Justice as a Step toward Racial Equity in School Discipline

The purpose of this multimedia research is to provide a blueprint for change that is centered on an alternative disciplinary approach referred to as restorative justice or restorative practices. First, we provide a short overview of the problem of racially based discipline practices in American schools. Then, we share the philosophical underpinnings of restorative justice, describe key components essential to its implementation, and provide links to videos that illustrate the successful implementation of restorative practices in authentic school settings. Thereafter, we offer what we believe is vital for institutional change: understanding the role Whiteness plays in disparate treatment and engaging in anti-racist school leadership. In the final section of the paper, we share specific strategies educators can use to navigate the change processes necessary to work toward racial equity in school discipline.

Fusion that matters

In recent years, deep learning has been successfully applied to diverse multimedia research areas, with the aim of learning powerful and informative representations for a variety of visual recognit...

OmniArt

Baselines are the starting point of any quantitative multimedia research, and benchmarks are essential for pushing those baselines further. In this article, we present baselines for the artistic domain with a new benchmark dataset featuring over 2 million images with rich structured metadata dubbed OmniArt. OmniArt contains annotations for dozens of attribute types and features semantic context information through concepts, IconClass labels, color information, and (limited) object-level bounding boxes. For our dataset we establish and present baseline scores on multiple tasks such as artist attribution, creation period estimation, type, style, and school prediction. In addition to our metadata related experiments, we explore the color spaces of art through different types and evaluate a transfer learning object recognition pipeline.