Photo Stream Research Articles

VISUAL MEDIA INTHE CONTEXT OF ZYGMUNT BAUMAN'S THEORY OF «LIQUID MODERNITY»

The article delves into the cultural-philosophical exploration of new visual media through Zygmunt Bauman's concept of «liquid modernity» (1925–2017), with a specific focus on the role of these media in shaping identity. The appeal of media is attributed to their pervasive presence in modern life, where smartphones are seen as extensions of the human body, and news feeds deliver a rapid, uncontrolled stream of photos and videos. Examining new media within the framework of liquid modernity can potentially reveal a significant correlation between these phenomena, complementing Bauman's concept of life strategies in this era. The article outlines Bauman's key strategies in influencing contemporary identity, detailing the archetypes of the «pilgrim», the «vagabond», the «tourist», and the «player». It also explores the fundamental differences in theory between modernity and postmodernity, highlighting the criticisms Bauman has of liquid modernity, such as family crises, political impotence, chaos, inconsistency, rampant consumerism, and virtuality. Bauman's perspective on media appears predominantly pessimistic, viewing the virtual space primarily as an escape from reality. Comparing the role of media in the concept of liquid modernity with seminal works in media theory (particularly the ideas of Marshall McLuhan and Friedrich Kittler) underscores the need to integrate media studies into Bauman's philosophy for a more profound understanding of identity construction principles in modernity. Analyzing McLuhan's approach to defining and analyzing media reveals that comprehending the societal and cultural changes brought about by emerging new media, which themselves convey messages, is crucial for understanding identity construction methods. McLuhan posits that the dominant culture type is shaped by the primary media of the era, suggesting that while printing led to a shift from audio-tactile to visual culture, the 20th century saw a resurgence of audio-tactile culture with the rise of new media. To further explore media's role as a tool for manipulating reality, as emphasized by Bauman, we can turn to Susan Sontag's work. In her essay collection «Regarding the Pain of Others», Sontag analyzes the portrayal of violence in the media.

Imagine, Reason and Write: Visual Storytelling with Graph Knowledge and Relational Reasoning

Visual storytelling is a task of creating a short story based on photo streams. Different from visual captions, stories contain not only factual descriptions, but also imaginary concepts that do not appear in the images. In this paper, we propose a novel imagine-reason-write generation framework (IRW) for visual storytelling, inspired by the logic of humans when they write the story. First, an imagine module is leveraged to learn the imaginative storyline explicitly, improving the coherence and reasonability of the generated story. Second, we employ a reason module to fully exploit the external knowledge (commonsense knowledge base) and task-specific knowledge (scene graph and event graph) with relational reasoning method based on the storyline. In this way, we can effectively capture the most informative commonsense and visual relationships among objects in images, which enhances the diversity and informativeness of the generated story. Finally, we integrate the imaginary concepts and relational knowledge to generate human-like story based on the original semantics of images. Extensive experiments on a benchmark dataset (i.e., VIST) demonstrate that the proposed IRW framework significantly outperforms the state-of-the-art methods across multiple evaluation metrics.

A hierarchical parallel fusion framework for egocentric ADL recognition based on discernment frame partitioning and belief coarsening

Recently, egocentric activity recognition has become a major research area in pattern recognition and artificial intelligence due to its high significance in potential applications in medical care, rehabilitation, smart home/office, etc. In this study, we develop a hierarchical parallel multimodal fusion framework for the recognition of egocentric activities in daily living (ADL). This framework uses the Dezert–Smarandache theory and is constructed around three modalities: location, motion and vision data from a wearable hybrid sensor system. The reciprocal distance and a trained support vector machine classifier are used to form the basic belief assignments (BBA) of location and motion. For vision data composed of egocentric photo streams, a well-trained convolutional neural network is utilized to produce a set of textual tags and the entropy-based statistics for these tags are used to construct the vision BBA. Discernment partitioning and belief coarsening theory are adopted for the hierarchical fusion of the three BBA functions from different ADL levels. Experimental results show that the recognition accuracy of the proposed fusion method was significantly higher than that of the methods based on single modality or modality combinations when our method was applied to real-life multimodal egocentric activity datasets. In addition, our method also achieved higher adaptability and scalability.

Hide-and-Tell: Learning to Bridge Photo Streams for Visual Storytelling

Visual storytelling is a task of creating a short story based on photo streams. Unlike existing visual captioning, storytelling aims to contain not only factual descriptions, but also human-like narration and semantics. However, the VIST dataset consists only of a small, fixed number of photos per story. Therefore, the main challenge of visual storytelling is to fill in the visual gap between photos with narrative and imaginative story. In this paper, we propose to explicitly learn to imagine a storyline that bridges the visual gap. During training, one or more photos is randomly omitted from the input stack, and we train the network to produce a full plausible story even with missing photo(s). Furthermore, we propose for visual storytelling a hide-and-tell model, which is designed to learn non-local relations across the photo streams and to refine and improve conventional RNN-based models. In experiments, we show that our scheme of hide-and-tell, and the network design are indeed effective at storytelling, and that our model outperforms previous state-of-the-art methods in automatic metrics. Finally, we qualitatively show the learned ability to interpolate storyline over visual gaps.

A Hierarchical Deep Fusion Framework for Egocentric Activity Recognition Using a Wearable Hybrid Sensor System.

Recently, egocentric activity recognition has attracted considerable attention in the pattern recognition and artificial intelligence communities because of its wide applicability in medical care, smart homes, and security monitoring. In this study, we developed and implemented a deep-learning-based hierarchical fusion framework for the recognition of egocentric activities of daily living (ADLs) in a wearable hybrid sensor system comprising motion sensors and cameras. Long short-term memory (LSTM) and a convolutional neural network are used to perform egocentric ADL recognition based on motion sensor data and photo streaming in different layers, respectively. The motion sensor data are used solely for activity classification according to motion state, while the photo stream is used for further specific activity recognition in the motion state groups. Thus, both motion sensor data and photo stream work in their most suitable classification mode to significantly reduce the negative influence of sensor differences on the fusion results. Experimental results show that the proposed method not only is more accurate than the existing direct fusion method (by up to 6%) but also avoids the time-consuming computation of optical flow in the existing method, which makes the proposed algorithm less complex and more suitable for practical application.

Show, Reward and Tell: Automatic Generation of Narrative Paragraph From Photo Stream by Adversarial Training

Impressive image captioning results (i.e., an objective description for an image) are achieved with plenty of training pairs. In this paper, we take one step further to investigate the creation of narrative paragraph for a photo stream. This task is even more challenging due to the difficulty in modeling an ordered photo sequence and in generating a relevant paragraph with expressive language style for storytelling. The difficulty can even be exacerbated by the limited training data, so that existing approaches almost focus on search-based solutions. To deal with these challenges, we propose a sequence-to-sequence modeling approach with reinforcement learning and adversarial training. First, to model the ordered photo stream, we propose a hierarchical recurrent neural network as story generator, which is optimized by reinforcement learning with rewards. Second, to generate relevant and story-style paragraphs, we design the rewards with two critic networks, including a multi-modal and a language-style discriminator. Third, we further consider the story generator and reward critics as adversaries. The generator aims to create indistinguishable paragraphs to human-level stories, whereas the critics aim at distinguishing them and further improving the generator by policy gradient. Experiments on three widely-used datasets show the effectiveness, against state-of-the-art methods with relative increase of 20.2% by METEOR. We also show the subjective preference for the proposed approach over the baselines through a user study with 30 human subjects.

‘What is puberty, then?’ Smartphones and Tumblr images as de/re-territorialisations in an upper secondary school classroom

ABSTRACTIn this paper, we explore the part mobile phone use plays in the capitalist assemblages present in school classrooms. Capitalism is approached through the continuous movement of de- and re-territorialisation. The empirical grounding is a wide study on mobile phone use conducted at an upper secondary school in Finland. The focus of this paper is one psychology lesson on puberty and the ways in which territories such as puberty are challenged in the school life of young people through their mobile phone use in class. Analysing a student’s Tumblr photo stream, we show how smartphones challenge the existing territorialities in a classroom. We locate three central deterritorialising movements: in relation to the physical space of the classroom; to the affective space of the classroom; and to the notion of body in puberty. We conclude that mobile phone use matters to students intimately and channels flows of capitalism at school.

Semi-supervised learning on large-scale geotagged photos for situation recognition

Photos are becoming spontaneous, objective, and universal sources of information. This paper explores evolving situation recognition using photo streams coming from disparate sources combined with the advances of deep learning. Using visual concepts in photos together with space and time information, we formulate the situation detection into a semi-supervised learning framework and propose new graph-based models to solve the problem. To extend the method for unknown situations, we introduce a soft label method that enables the traditional semi-supervised learning framework to accurately predict predefined labels as well as effectively form new clusters. To overcome the noisy data which degrades graph quality, leading to poor recognition results, we take advantage of two kinds of noise-robust norms which can eliminate the adverse effects of outliers in visual concepts and improve the accuracy of situation recognition. Finally, we demonstrate the idea and the effectiveness of the proposed models on Yahoo Flickr Creative Commons 100 Million.

Retrieval of Sentence Sequences for an Image Stream via Coherence Recurrent Convolutional Networks.

We propose an approach for retrieving a sequence of natural sentences for an image stream. Since general users often take a series of pictures on their experiences, much online visual information exists in the form of image streams, for which it would better take into consideration of the whole image stream to produce natural language descriptions. While almost all previous studies have dealt with the relation between a single image and a single natural sentence, our work extends both input and output dimension to a sequence of images and a sequence of sentences. For retrieving a coherent flow of multiple sentences for a photo stream, we propose a multimodal neural architecture called coherence recurrent convolutional network (CRCN), which consists of convolutional neural networks, bidirectional long short-term memory (LSTM) networks, and an entity-based local coherence model. Our approach directly learns from vast user-generated resource of blog posts as text-image parallel training data. We collect more than 22 K unique blog posts with 170 K associated images for the travel topics of NYC, Disneyland , Australia, and Hawaii. We demonstrate that our approach outperforms other state-of-the-art image captioning methods for text sequence generation, using both quantitative measures and user studies via Amazon Mechanical Turk.

Let Your Photos Talk: Generating Narrative Paragraph for Photo Stream via Bidirectional Attention Recurrent Neural Networks

Automatic generation of natural language description for individual images (a.k.a. image captioning) has attracted extensive research attention. In this paper, we take one step further to investigate the generation of a paragraph to describe a photo stream for the purpose of storytelling. This task is even more challenging than individual image description due to the difficulty in modeling the large visual variance in an ordered photo collection and in preserving the long-term language coherence among multiple sentences. To deal with these challenges, we formulate the task as a sequence-to-sequence learning problem and propose a novel joint learning model by leveraging the semantic coherence in a photo stream. Specifically, to reduce visual variance, we learn a semantic space by jointly embedding each photo with its corresponding contextual sentence, so that the semantically related photos and their correlations are discovered. Then, to preserve language coherence in the paragraph, we learn a novel Bidirectional Attention-based Recurrent Neural Network (BARNN) model, which can attend on the discovered semantic relation to produce a sentence sequence and maintain its consistence with the photo stream. We integrate the two-step learning components into one single optimization formulation and train the network in an end-to-end manner. Experiments on three widely-used datasets (NYC/Disney/SIND) show that the proposed approach outperforms state-of-the-art methods with large margins for both retrieval and paragraph generation tasks. We also show the subjective preference of the machine-generated stories by the proposed approach over the baselines through a user study with 40 human subjects.

SR-clustering: Semantic regularized clustering for egocentric photo streams segmentation

While wearable cameras are becoming increasingly popular, locating relevant information in large unstructured collections of egocentric images is still a tedious and time consuming process. This paper addresses the problem of organizing egocentric photo streams acquired by a wearable camera into semantically meaningful segments, hence making an important step towards the goal of automatically annotating these photos for browsing and retrieval. In the proposed method, first, contextual and semantic information is extracted for each image by employing a Convolutional Neural Networks approach. Later, a vocabulary of concepts is defined in a semantic space by relying on linguistic information. Finally, by exploiting the temporal coherence of concepts in photo streams, images which share contextual and semantic attributes are grouped together. The resulting temporal segmentation is particularly suited for further analysis, ranging from event recognition to semantic indexing and summarization. Experimental results over egocentric set of nearly 31,000 images, show the prominence of the proposed approach over state-of-the-art methods.

Multiple Human Identification and Cosegmentation: A Human-Oriented CRF Approach With Poselets

Localizing, identifying, and extracting humans with consistent appearance jointly from a personal photo stream is an important problem and has wide applications. The strong variations in foreground and background and irregularly occurring foreground humans make this realistic problem challenging. Inspired by advancements in object detection, scene understanding, and image cosegmentation, we explore explicit constraints to label and segment human objects rather than other nonhuman objects and “stuff.” We refer to such a problem as multiple human identification and cosegmentation (MHIC). To identify specific human subjects, we propose an efficient human instance detector by combining an extended color line model with a poselet-based human detector. Moreover, to capture high-level human shape information, a novel soft shape cue is proposed. It is initialized by the human detector, then further enhanced through a generalized geodesic distance transform, and finally refined with a joint bilateral filter. We also propose to capture the rich feature context around each pixel by using an adaptive cross-region data structure, which gives a higher discriminative power than a single pixel-based estimation. The high-level object cues from the detector and the shape are then integrated with the low-level pixel cues and midlevel contour cues into a principled conditional random field (CRF) framework, which can be efficiently solved by using fast graph cut algorithms. We evaluate our method over a newly created NTU-MHIC human dataset, which contains 351 images with manually annotated groundtruth segmentation. Both visual and quantitative results demonstrate that our method achieves state-of-the-art performance for the MHIC task.

효과적인 이미지 검색을 위한 태그 기반의 폭소노미 이미지 카테고리화 기법

최근 사용자들이 협동적으로 이미지 주석인 태그를 만들고 활용하는 폭소노미 기반의 이미지공유 사이트들이 많은 인기를 얻고 있다. 이러한 사이트는 사용자 질의에 대해 단순한 텍스트 매칭 기반의 검색을 수행하고 매칭되는 결과 이미지들을 포토 스트림 형태로 나열하여 보여 준다. 하지만 이러한 태그들은 매우 개인적이고 주관적이며, 이미지 역시 카테고리로 분류되어 있지 않기 때문에 검색의 정확도나 사용자 만족도가 떨어진다는 문제점이 있다. 본 연구에서는 태그를 기반으로 하는 이미지 검색에서 검색의 정확도를 높일 수 있는 폭소노미 이미지의 카테고리화 기법을 제안하고, 폭소노미 환경에서 생성된 태그와 이미지 정보를 모두 이용하며 의미적으로 유사한 이미지들끼리 분류된 검색 결과를 생성한다. 제안하는 기법의 성능 평가를 위해 폭소노미 이미지를 수집하고 텍스트, 이미지 특성에 따른 카테고리 분류를 수행하여 기존 검색 기법과 이미지 검색의 정확도를 비교한다.

Adolescents' presentation of food in social media: An explorative study

The study aimed to explore how adolescents communicate food images in a widely used social media image-sharing application. We examined how and in what context food was presented and the type of food items that were frequently portrayed by following a youth related hashtag on Instagram. The hashtag #14år (“14 years”) was used to find adolescent users on Instagram: these users public photo streams were then searched for food items they had shared with others. Food items were identified and categorized based on type of food and how the food items were presented. Most of the adolescent users (85%) shared images containing food items. A majority of the images (67.7%) depicted foods high in calories but low in nutrients. Almost half of these images were arranged as a still life with food brand names clearly exposed. Many of these images were influenced by major food marketing campaigns. Fruits and vegetables occurred in 21.8% of all images. This food group was frequently portrayed zoomed in with focus solely on the food, with a hashtag or caption expressing palatability. These images were often presented in the style of a cook book. Food was thus presented in varied ways. Adolescents themselves produced images copying food advertisements. This has clear health promotion implications since it becomes more challenging to monitor and tackle young people's exposure to marketing of unhealthy foods in these popular online networks because images are part of a lifestyle that the young people want to promote. Shared images contain personal recommendations, which mean that they may have a more powerful effect than commercial advertising.

Introduction to Hélio Oiticica’s “Héliotape with Mario Montez (1971)”

Born in Rio de Janeiro in 1937, Brazilian artist Helio Oiticica lived in New York City in 1971-78. As did many political exiles and other fellow artists abroad during the years of military dictatorship in Brazil, Oiticica would help himself through difficult times with a bit of cocaine dealing here and there. This enterprise provided him with a remarkable social mobility. Oiticica soon had friends on Street and in rock-star circles, highbrow bohemia, and the queer underground. At the same time, he became a sort of chronicler of life in Lower Manhattan-MoMA (Museum of Modern Art) artist and occasional St. Marks Place pusher at once-and provided his friends and various clandestine countercultural magazines back in Brazil with a ceaseless stream of letters, articles, photos, films, and tape recordings. In these, he would mix such heterogeneous elements as anecdotes of his life in the queer underground, detailed drug and cruising experiences, metropolitan gossip, philosophical reading advice, and comments on film, music, and art scenes. (I don't know what is going on here, there is such a bourgeois art scene, conformity and reactionarism going on, unbelievable!1)The name Heliotapes describes a series of tape recordings that Oiticica made in this fashion between 1971 and 1975. The following is a transcription of one particular Heliotape that Oiticica recorded in conversation with Montez in 1971. By then, Oiticica had reached the conclusion that the sellout of New York's queer underground had long begun, a fact that was confirmed by the success of Andy Warhol-Paul Morrissey film Trash (1970) in what he regarded as neoconservative circles that voyeuristically feed on marginal activity. Against this logic of spectacle and consumption, Oiticica maintained a particularly strong affinity to Montez and Jack Smith, whom he considered to be different. Thus, one year after their recorded conversation, Montez would appear in the most famous of Oiticica's Super 8 films, Agripina e Roma Manhattan {Agrippina Is Rome-Manhattan, 1972), in which he and Brazilian artist Antonio Dias throw dice, evoking what Oiticica called a Wall Street Oracle or OraCULO.Already in 1971 Oiticica had used the Heliotape conversation as the basis for his article Mario Montez, Tropicamp that he published in Rio de Janeiro's Presenca magazine.2 In this text, Oiticica envisioned a sort of internationalist underground solidarity between what once had been the notorious Tropicalia movement in Brazil-but now was scattered and fragmented over South America, Europe, and the United States-and Smith and Montez, representing the element resistant to both commercialization and the distinctive chic of New York's post-1968 bohemia. …

Structure Preserving Large Imagery Reconstruction

With the explosive growth of web-based cameras and mobile devices, billions of photographs are uploaded to the internet. We can trivially collect a huge number of photo streams for various goals, such as image clustering, 3D scene reconstruction, and other big data applications. However, such tasks are not easy due to the fact the retrieved photos can have large variations in their view perspectives, resolutions, lighting, noises , and distortions. Furthermore, with the occlusion of unexpected objects like people, vehicles, it is even more challenging to find feature correspondences and reconstruct realistic scenes. In this paper, we propose a structure-based image completion algorithm for object removal that produces visually plausible content with consistent structure and scene texture. We use an edge matching technique to infer the potential structure of the unknown region. Driven by the estimated structure, texture synthesis is performed automatically a long the estimated curves. We evaluate the proposed method on different types of images: from

Photo Stream Alignment and Summarization for Collaborative Photo Collection and Sharing

With the popularity of digital cameras and camera phones, it is common for different people, who may or may not know each other, to attend the same event and take pictures and videos from different spatial or personal perspectives. Within the realm of social media, it is desirable to enable these people to select and share their pictures and videos in order to enrich memories and facilitate social networking. However, it is cumbersome to manually manage these photos from different cameras, of which the clocks settings are often not calibrated. In this paper, we propose automatic algorithms to address the above problems. First, we accurately align different photo streams or sequences from different photographers for the same event in chronological order on a common timeline, while respecting the time constraints within each photo stream. Given the preferred similarity measures (e.g., visual, and spatial similarities), our algorithm performs photo stream alignment via matching on a bipartite kernel sparse representation graph that forces the data connections to be sparse in an explicit fashion. Furthermore, we can produce a summary master stream from the aligned super stream of photos for efficient sharing by removing those redundant photos in the super stream while accounting for the temporal integrity. Based on a similar kernel sparse representation graph, our master stream summarization algorithm performs greedy backward selection to drop redundant photos without affecting the integrity of remaining photos for the entire event. We evaluate our algorithms on real-world personal online albums for 36 events and demonstrate its efficacy in automatically facilitating collaborative photo collection and sharing.

Multimedia Semantics: Interactions Between Content and Community

This paper reviews the state of the art and some emerging issues in research areas related to pattern analysis and monitoring of web-based social communities. This research area is important for several reasons. First, the presence of near-ubiquitous low-cost computing and communication technologies has enabled people to access and share information at an unprecedented scale. The scale of the data necessitates new research for making sense of such content. Furthermore, popular websites with sophisticated media sharing and notification features allow users to stay in touch with friends and loved ones; these sites also help to form explicit and implicit social groups. These social groups are an important source of information to organize and to manage multimedia data. In this article, we study how media-rich social networks provide additional insight into familiar multimedia research problems, including tagging and video ranking. In particular, we advance the idea that the contextual and social aspects of media are as important for successful multimedia applications as is the media content. We examine the inter-relationship between content and social context through the prism of three key questions. First, how do we extract the context in which social interactions occur? Second, does social interaction provide value to the media object? Finally, how do social media facilitate the repurposing of shared content and engender cultural memes? We present three case studies to examine these questions in detail. In the first case study, we show how to discover structure latent in the social media data, and use the discovered structure to organize Flickr photo streams. In the second case study, we discuss how to determine the interestingness of conversations—and of participants—around videos uploaded to YouTube. Finally, we show how the analysis of visual content, in particular tracing of content remixes, can help us understand the relationship among YouTube participants. For each case, we present an overview of recent work and review the state of the art. We also discuss two emerging issues related to the analysis of social networks—robust data sampling and scalable data analysis.

Discovering multirelational structure in social media streams

In this article, we present a novel algorithm to discover multirelational structures from social media streams. A media item such as a photograph exists as part of a meaningful interrelationship among several attributes, including time, visual content, users, and actions. Discovery of such relational structures enables us to understand the semantics of human activity and has applications in content organization, recommendation algorithms, and exploratory social network analysis. We are proposing a novel nonnegative matrix factorization framework to characterize relational structures of group photo streams. The factorization incorporates image content features and contextual information. The idea is to consider a cluster as having similar relational patterns; each cluster consists of photos relating to similar content or context. Relations represent different aspects of the photo stream data, including visual content, associated tags, photo owners, and post times. The extracted structures minimize the mutual information of the predicted joint distribution. We also introduce a relational modularity function to determine the structure cost penalty, and hence determine the number of clusters. Extensive experiments on a large Flickr dataset suggest that our approach is able to extract meaningful relational patterns from group photo streams. We evaluate the utility of the discovered structures through a tag prediction task and through a user study. Our results show that our method based on relational structures, outperforms baseline methods, including feature and tag frequency based techniques, by 35%--420%. We have conducted a qualitative user study to evaluate the benefits of our framework in exploring group photo streams. The study indicates that users found the extracted clustering results clearly represent major themes in a group; the clustering results not only reflect how users describe the group data but often lead the users to discover the evolution of the group activity.

Living Avatars Network: Fusing traditional and innovative ethnographic methods through a real‐time mobile video service

This paper presents a study of new technologies potentially enabling access to a sensory feast of places by ‘wired up’ flaněurs, real‐time as well as remote ‘native’ description and interactions and situated oral histories excavated through ‘being in a place’. We describe an inter‐disciplinary research project examining the cultural heritage of Singapore and the use of geo‐location technologies incorporating social networking platforms as a medium for interactive heritage walks. The goal of the project is to engage both locals and non‐locals in experiencing Singapore from a first person perspective, giving them a wider understanding of the ethnic and cultural diversity. The Living Avatar Network (LAN) supports sharing experiences and realities in real time through making it possible to ‘walk in someone's shoes’ through a living avatar, re‐experiencing someone's memories of a certain place. Here we describe the approaches deployed in evolving a prototypical service –‘traditional’ ethnographic style methods with the access to the real‐time, lived character of ‘local’ experiences offered by digital photo streams and real‐time video. More broadly, the project acknowledges the potential availability of the experience of being in a place and culture to be widely available through Web 2.0 technologies and people spending more time ‘living digitally’.