Egocentric Images Research Articles

Egocentric Performance Capture: A Review

Performance capture of human beings have been used to animate 3D characters for movies and games for several decades now. Traditional performance capture methods require a dedicated costly setup which usually consists of more than one sensor placed at a distance from the subject, hence requiring a large amount of budget and space to accomodate. This lowers its feasibility and portability by a huge amount. Egocentric (first-person/wearable) cameras, however, are attached to the body and hence are mobile. With a rise of acceptance of wearable technology by the general public, wearable cameras have gotten cheaper too. We can make use of their excessive portability in the performance capture domain. However working with egocentric images is a mammoth task as the views are severely distorted due to the first-person perspective and the body parts farther from the camera are highly prone to being occluded. In this paper, we review the existing state-of-the-art methods about performance capture using egocentric based views.

Egocentric visitors localization in natural sites

Localizing visitors in natural environments is challenging due to the unavailability of pre-installed cameras or other infrastructure such as WiFi networks. We propose to perform localization using egocentric images collected from the visitor’s point of view with a wearable camera. Localization can be useful to provide services to both the visitors (e.g., showing where they are or what to see next) and to the site manager (e.g., to understand what the visitors pay more attention to and what they miss during their visits). We collected and publicly released a dataset of egocentric videos asking 12 subjects to freely visit a natural site. Along with video, we collected GPS locations by means of a smartphone. Experiments comparing localization methods based on GPS and images highlight that image-based localization is much more reliable in the considered domain and small improvements can be achieved by combining GPS- and image-based predictions using late fusion.

Egocentric Visitors Localization in Cultural Sites

We consider the problem of localizing visitors in a cultural site from egocentric (first-person) images. Localization information can be useful both to assist the user during his visit (e.g., by suggesting where to go and what to see next) and to provide behavioral information to the manager of the cultural site (e.g., how much time has been spent by visitors at a given location? What has been liked most?). To tackle the problem, we collected a large dataset of egocentric videos using two cameras: a head-mounted HoloLens device and a chest-mounted GoPro. Each frame has been labeled according to the location of the visitor and to what he was looking at. The dataset is freely available in order to encourage research in this domain. The dataset is complemented with baseline experiments performed considering a state-of-the-art method for location-based temporal segmentation of egocentric videos. Experiments show that compelling results can be achieved to extract useful information for both the visitor and the site-manager.

Differential influence of habitual third-person vision of a body part on mental rotation of images of hands and feet.

Left/right judgement (LRJ) tasks involve determining the laterality of presented hand or feet images. Allocentric images (third-person perspective; 3PP) take longer to identify than egocentric images (first-person perspective; 1PP), supporting that implicit motor imagery (IMI)-mentally manoeuvring one's body to match the shown posture-is used. While numerous cognitive processes are involved during LRJs, it remains unclear whether features of the individual (e.g., visual exposure, experience, task-dependent use) influence the type of recognition strategy used during LRJs (IMI versus non-IMI). To investigate whether an individual's routine visual exposure to hands/feet in 3PP disrupts the typical perspective-reaction time (RT) relationship in LRJs, hand therapists, podiatrists, and healthy controls completed online LRJ tasks of hand and feet images. A group-specific reduction in RT for only allocentric images would represent a switch to non-IMI strategies. The results show that routine visual exposure to feet in 3PP (podiatrists) results in quicker RTs only for allocentric images of feet, suggesting a switch from IMI to non-IMI (e.g., visual object-based recognition) strategies. In contrast, routine visual exposure to hands in 3PP (hand therapists) does not alter RT for allocentric images, suggesting maintenance of IMI. However, hand therapists have quicker RTs (vs other groups) for egocentric hand images, supporting enhanced sensorimotor processing for the hand, consistent with task-dependent use (precise hand use). Higher accuracy in health professionals (vs control) on both tasks supports enhanced body schema. Combined, this suggests that 3PP visual exposure to body parts and task-dependent use contribute to LRJ performance/recognition strategy.

Un-supervised and semi-supervised hand segmentation in egocentric images with noisy label learning

With the rapid development of wearable devices and technologies, hand segmentation remains a less explored direction in egocentric vision, which is very important for activity recognition, rehabilitation, robot self-learning, etc. To overcome the high cost of auxiliary equipment and pixel-level annotations, we present an un-supervised hand segmentation method for egocentric images. Firstly, a fully convolutional neural network (FCN) is pre-trained in source dataset containing pixel-level annotations. Then, in target dataset without labels, the network is re-trained with optimized masks, which are produced by modified local and global consistency learning (LLGC) based on pre-segmentation and superpixel features. Finally, hand segmentation is realized in an alternative way. Furthermore, to balance segmentation accuracy and the cost on labeling, we propose a new semi-supervised image segmentation framework with three sub-nets based on the optimized noisy masks and a small number of clean labeled data. Experimental results in two target datasets indicate that the proposed methods could achieve better performance than other methods.

Automatic food detection in egocentric images using artificial intelligence technology.

To develop an artificial intelligence (AI)-based algorithm which can automatically detect food items from images acquired by an egocentric wearable camera for dietary assessment. To study human diet and lifestyle, large sets of egocentric images were acquired using a wearable device, called eButton, from free-living individuals. Three thousand nine hundred images containing real-world activities, which formed eButton data set 1, were manually selected from thirty subjects. eButton data set 2 contained 29 515 images acquired from a research participant in a week-long unrestricted recording. They included both food- and non-food-related real-life activities, such as dining at both home and restaurants, cooking, shopping, gardening, housekeeping chores, taking classes, gym exercise, etc. All images in these data sets were classified as food/non-food images based on their tags generated by a convolutional neural network. A cross data-set test was conducted on eButton data set 1. The overall accuracy of food detection was 91·5 and 86·4 %, respectively, when one-half of data set 1 was used for training and the other half for testing. For eButton data set 2, 74·0 % sensitivity and 87·0 % specificity were obtained if both 'food' and 'drink' were considered as food images. Alternatively, if only 'food' items were considered, the sensitivity and specificity reached 85·0 and 85·8 %, respectively. The AI technology can automatically detect foods from low-quality, wearable camera-acquired real-world egocentric images with reasonable accuracy, reducing both the burden of data processing and privacy concerns.

Egocentric video description based on temporally-linked sequences

Egocentric vision consists in acquiring images along the day from a first person point-of-view using wearable cameras. The automatic analysis of this information allows to discover daily patterns for improving the quality of life of the user. A natural topic that arises in egocentric vision is storytelling, that is, how to understand and tell the story relying behind the pictures.In this paper, we tackle storytelling as an egocentric sequences description problem. We propose a novel methodology that exploits information from temporally neighboring events, matching precisely the nature of egocentric sequences. Furthermore, we present a new method for multimodal data fusion consisting on a multi-input attention recurrent network. We also release the EDUB-SegDesc dataset. This is the first dataset for egocentric image sequences description, consisting of 1339 events with 3991 descriptions, from 55 days acquired by 11 people. Finally, we prove that our proposal outperforms classical attentional encoder-decoder methods for video description.

SR-clustering: Semantic regularized clustering for egocentric photo streams segmentation

While wearable cameras are becoming increasingly popular, locating relevant information in large unstructured collections of egocentric images is still a tedious and time consuming process. This paper addresses the problem of organizing egocentric photo streams acquired by a wearable camera into semantically meaningful segments, hence making an important step towards the goal of automatically annotating these photos for browsing and retrieval. In the proposed method, first, contextual and semantic information is extracted for each image by employing a Convolutional Neural Networks approach. Later, a vocabulary of concepts is defined in a semantic space by relying on linguistic information. Finally, by exploiting the temporal coherence of concepts in photo streams, images which share contextual and semantic attributes are grouped together. The resulting temporal segmentation is particularly suited for further analysis, ranging from event recognition to semantic indexing and summarization. Experimental results over egocentric set of nearly 31,000 images, show the prominence of the proposed approach over state-of-the-art methods.

Multi-face tracking by extended bag-of-tracklets in egocentric photo-streams

Wearable cameras offer a hands-free way to record egocentric images of daily experiences, where social events are of special interest. The first step towards detection of social events is to track the appearance of multiple persons involved in them. In this paper, we propose a novel method to find correspondences of multiple faces in low temporal resolution egocentric videos acquired through a wearable camera. This kind of photo-stream imposes additional challenges to the multi-tracking problem with respect to conventional videos. Due to the free motion of the camera and to its low temporal resolution, abrupt changes in the field of view, in illumination condition and in the target location are highly frequent. To overcome such difficulties, we propose a multi-face tracking method that generates a set of tracklets through finding correspondences along the whole sequence for each detected face and takes advantage of the tracklets redundancy to deal with unreliable ones. Similar tracklets are grouped into the so called extended bag-of-tracklets (eBoT), which is aimed to correspond to a specific person. Finally, a prototype tracklet is extracted for each eBoT, where the occurred occlusions are estimated by relying on a new measure of confidence. We validated our approach over an extensive dataset of egocentric photo-streams and compared it to state of the art methods, demonstrating its effectiveness and robustness.

Predicting Daily Activities From Egocentric Images Using Deep Learning.

We present a method to analyze images taken from a passive egocentric wearable camera along with the contextual information, such as time and day of week, to learn and predict everyday activities of an individual. We collected a dataset of 40,103 egocentric images over a 6 month period with 19 activity classes and demonstrate the benefit of state-of-the-art deep learning techniques for learning and predicting daily activities. Classification is conducted using a Convolutional Neural Network (CNN) with a classification method we introduce called a late fusion ensemble. This late fusion ensemble incorporates relevant contextual information and increases our classification accuracy. Our technique achieves an overall accuracy of 83.07% in predicting a person's activity across the 19 activity classes. We also demonstrate some promising results from two additional users by fine-tuning the classifier with one day of training data.

The hippocampus and spatial constraints on mental imagery

We review a model of imagery and memory retrieval based on allocentric spatial representation by place cells and boundary vector cells (BVCs) in the medial temporal lobe, and their translation into egocentric images in retrosplenial and parietal areas. In this model, the activity of place cells constrain the contents of imagery and retrieval to be coherent and consistent with the subject occupying a single location, while the activity of head-direction cells along Papez's circuit determine the viewpoint direction for which the egocentric image is generated. An extension of this model is discussed in which a role for grid cells in dynamic updating of representations (mental navigation) is included. We also discuss the extension of this model to implement a version of the dual representation theory of post-traumatic stress disorder (PTSD) in which PTSD arises from an imbalance between weak allocentric hippocampal-mediated contextual representations and strong affective/sensory representations. The implications of these models for behavioral, neuropsychological, and neuroimaging data in humans are explored.

The emergence of social psychology in Vienna: An exercise in long‐term memory*

For the reconstruction of the past eyewitnesses are as important as they are suspect. What happened, what one thought and felt more than half a century ago, is, in part, irrevocably lost; what remains alive in memory are kaleidoscopic, discontinuous and egocentric images. What really happened appeared differently to different participants even then; all the more so now. Going beyond the images to a coherent presentation requires a contemporary reconstruction, inevitably influenced by intervening events, by the well‐known vicissitudes of long‐term memory (selectivity, importation, distortion and repression), and by the purpose of a presentation. My aim here is to describe the atmosphere of those long‐ago years as faithfully as I can. Two other accounts of the emergence of an empirically based social science in Vienna had somewhat different intentions. Paul Lazarsfeld (1968), the dominant mind in this development, and Hans Zeisel (1968) have recalled these early beginnings in relation to their own subsequent work and achievements. The following remarks are complementary to theirs. Since they take off from memory images, they will inevitably be less systematic and more personal than theirs, though there are, of course, considerable overlaps as well as some differences in interpretation.