Egocentric View Research Articles

Fusion-GRU: A Deep Learning Model for Future Bounding Box Prediction of Traffic Agents in Risky Driving Videos

To ensure the safe and efficient navigation of autonomous vehicles and advanced driving assistance systems in complex traffic scenarios, predicting the future bounding boxes of surrounding traffic agents is crucial. However, simultaneously predicting the future location and scale of target traffic agents from the egocentric view poses challenges because the vehicles’ egomotion causes considerable field-of-view changes. Moreover, in anomalous or risky situations, tracking loss or abrupt motion changes limit the available observation time, requiring learning of cues within a short time window. Existing methods typically use a simple concatenation operation to combine different cues, overlooking their dynamics over time. To address this, this paper introduces the Fusion-Gated Recurrent Unit (Fusion-GRU) network, a novel encoder-decoder architecture for future bounding box localization. Unlike traditional gated recurrent units (GRUs), Fusion-GRU accounts for mutual and complex interactions among input features. Moreover, an intermediary estimator coupled with a self-attention aggregation layer is also introduced to learn sequential dependencies for long-range prediction. Finally, a GRU decoder is employed to predict the future bounding boxes. The proposed method is evaluated on two publicly available datasets, Risky Object Localization and Honda Egocentric Vehicle Intersection. The experimental results showcase the promising performance of the Fusion-GRU, demonstrating its effectiveness in predicting future bounding boxes of traffic agents.

Exocentric and Egocentric Views for Biomedical Data Analytics in Virtual Environments-A Usability Study.

Biomedical datasets are usually large and complex, containing biological information about a disease. Computational analytics and the interactive visualisation of such data are essential decision-making tools for disease diagnosis and treatment. Oncology data models were observed in a virtual reality environment to analyse gene expression and clinical data from a cohort of cancer patients. The technology enables a new way to view information from the outside in (exocentric view) and the inside out (egocentric view), which is otherwise not possible on ordinary displays. This paper presents a usability study on the exocentric and egocentric views of biomedical data visualisation in virtual reality and their impact on usability on human behaviour and perception. Our study revealed that the performance time was faster in the exocentric view than in the egocentric view. The exocentric view also received higher ease-of-use scores than the egocentric view. However, the influence of usability on time performance was only evident in the egocentric view. The findings of this study could be used to guide future development and refinement of visualisation tools in virtual reality.

Examination of Human Spatial Reasoning Capability and Simulated Autonomous Rendezvous and Docking Monitoring Performance

Autonomous rendezvous and docking (ARD) maneuvers are challenging tasks that require collaboration between a human and a spacecraft to be successful. As automation becomes more integrated into ARD systems, it is important to consider when and why a human may take control. Intrinsic human characteristics can influence these decisions. We consider how human spatial orientation capacity affects participants when monitoring a simulated ARD maneuver and initiating takeover when the system is perceived to be failing. Participants’ spatial reasoning capability was assessed and compared to performance in the monitoring task and perceived mental workload. While participants showed high rates of success in the task, they showed a wide range in spatial reasoning capacity and perceived mental demand. Spatial reasoning capacity did not indicate participants’ mental workload, which has implications for the human as the supervisor. These results inform future work on augmentative displays that may incorporate exocentric and egocentric views.

Goal-Directed Travel in the Nocturnal Javan Slow Loris (Nycticebus javanicus)

An animal’s ability to navigate its home range in search of essential resources is a key aspect of its ecology. To reach these resources, animals employ varying navigational processes depending on their exocentric or egocentric view of their environment. The goal of this study was to determine if the Javan slow loris (Nycticebus javanicus), a nocturnal arboreal primate found in southeast Asia, uses some form of cognitive map and spatial memory while navigating their environment. Using behavioural and GPS data of six males and seven females collected at the Little Fireface Project field station based in West Java, Indonesia, we measured their frequency of revisiting important feeding trees, route overlap, and points where individuals significantly changed directions. We found that all individuals predominantly used four tree species while feeding and foraging. The lorises also displayed a high level of route overlap, leading us to conclude that they likely utilize a route-based cognitive map where certain landmarks are integral to their nightly movement. Few studies have specifically focused on strepsirrhine spatial cognition in the wild; here, we show the navigational mechanism used by the Javan slow loris to reach distant/out-of-sight resources. The evident reliance on spatial cognition in a strepsirrhine species suggest that it could be an important selective pressure for primates at the earliest stages of primate cognitive evolution. In addition to the importance of spatial memory in theoretical discourse, understanding slow loris movement has practical applications to conservation, particularly regarding the numerous translocations undertaken by individuals rescued from the illegal wildlife trade. We discuss the importance of considering soft release training and monitoring in such releases.

Understanding First-Person and Third-Person Videos in Computer Vision

Due to advancements in technology and social media, a large amount of visual information is created. There is a lot of interesting research going on in Computer Vision that takes into consideration either visual information generated by first-person (egocentric) or third-person(exocentric) cameras. Video data generated by YouTubers, Surveillance cameras, and Drones which is referred to as third-person or exocentric video data. Whereas first-person or egocentric is the one which is generated by GoPro cameras and Google Glass. Exocentric view capture wide and global views whereas egocentric view capture activities an actor is involved in w.r.t. objects. These two perspectives seem to be independent yet related. In Computer Vision, these two perspectives have been studied by various domains like Activity Recognition, Object Detection, Action Recognition, and Summarization independently. Their relationship and comparison are less discussed in the literature. This paper tries to bridge this gap by presenting a systematic study of first-person and third-person videos. Further, we implemented an algorithm to classify videos as first-person/third-person with the validation accuracy of 88.4% and an F1-score of 86.10% using the Charades dataset..

Behavior-Aware Pedestrian Trajectory Prediction in Ego-Centric Camera Views with Spatio-Temporal Ego-Motion Estimation

With the ongoing development of automated driving systems, the crucial task of predicting pedestrian behavior is attracting growing attention. The prediction of future pedestrian trajectories from the ego-vehicle camera perspective is particularly challenging due to the dynamically changing scene. Therefore, we present Behavior-Aware Pedestrian Trajectory Prediction (BA-PTP), a novel approach to pedestrian trajectory prediction for ego-centric camera views. It incorporates behavioral features extracted from real-world traffic scene observations such as the body and head orientation of pedestrians, as well as their pose, in addition to positional information from body and head bounding boxes. For each input modality, we employed independent encoding streams that are combined through a modality attention mechanism. To account for the ego-motion of the camera in an ego-centric view, we introduced Spatio-Temporal Ego-Motion Module (STEMM), a novel approach to ego-motion prediction. Compared to the related works, it utilizes spatial goal points of the ego-vehicle that are sampled from its intended route. We experimentally validated the effectiveness of our approach using two datasets for pedestrian behavior prediction in urban traffic scenes. Based on ablation studies, we show the advantages of incorporating different behavioral features for pedestrian trajectory prediction in the image plane. Moreover, we demonstrate the benefit of integrating STEMM into our pedestrian trajectory prediction method, BA-PTP. BA-PTP achieves state-of-the-art performance on the PIE dataset, outperforming prior work by 7% in MSE-1.5 s and CMSE as well as 9% in CFMSE.

Learning the Vector Coding of Egocentric Boundary Cells from Visual Data.

The use of spatial maps to navigate through the world requires a complex ongoing transformation of egocentric views of the environment into position within the allocentric map. Recent research has discovered neurons in retrosplenial cortex and other structures that could mediate the transformation from egocentric views to allocentric views. These egocentric boundary cells respond to the egocentric direction and distance of barriers relative to an animal's point of view. This egocentric coding based on the visual features of barriers would seem to require complex dynamics of cortical interactions. However, computational models presented here show that egocentric boundary cells can be generated with a remarkably simple synaptic learning rule that forms a sparse representation of visual input as an animal explores the environment. Simulation of this simple sparse synaptic modification generates a population of egocentric boundary cells with distributions of direction and distance coding that strikingly resemble those observed within the retrosplenial cortex. Furthermore, some egocentric boundary cells learnt by the model can still function in new environments without retraining. This provides a framework for understanding the properties of neuronal populations in the retrosplenial cortex that may be essential for interfacing egocentric sensory information with allocentric spatial maps of the world formed by neurons in downstream areas, including the grid cells in entorhinal cortex and place cells in the hippocampus.SIGNIFICANCE STATEMENT The computational model presented here demonstrates that the recently discovered egocentric boundary cells in retrosplenial cortex can be generated with a remarkably simple synaptic learning rule that forms a sparse representation of visual input as an animal explores the environment. Additionally, our model generates a population of egocentric boundary cells with distributions of direction and distance coding that strikingly resemble those observed within the retrosplenial cortex. This transformation between sensory input and egocentric representation in the navigational system could have implications for the way in which egocentric and allocentric representations interface in other brain areas.

Evaluating the Effects of Virtual Reality Environment Learning on Subsequent Robot Teleoperation in an Unfamiliar Building.

Using a map in an unfamiliar environment requires identifying correspondences between elements of the map's allocentric representation and elements in egocentric views. Aligning the map with the environment can be challenging. Virtual reality (VR) allows learning about unfamiliar environments in a sequence of egocentric views that correspond closely to the perspectives and views that are experienced in the actual environment. We compared three methods to prepare for localization and navigation tasks performed by teleoperating a robot in an office building: studying a floor plan of the building and two forms of VR exploration. One group of participants studied a building plan, a second group explored a faithful VR reconstruction of the building from a normal-sized avatar's perspective, and a third group explored the VR from a giant-sized avatar's perspective. All methods contained marked checkpoints. The subsequent tasks were identical for all groups. The self-localization task required indication of the approximate location of the robot in the environment. The navigation task required navigation between checkpoints. Participants took less time to learn with the giant VR perspective and with the floorplan than with the normal VR perspective. Both VR learning methods significantly outperformed the floorplan in the orientation task. Navigation was performed quicker after learning in the giant perspective compared to the normal perspective and the building plan. We conclude that the normal perspective and especially the giant perspective in VR are viable options for preparing for teleoperation in unfamiliar environments when a virtual model of the environment is available.

Video parameters for action observation training in stroke rehabilitation: a scoping review

Purpose Action observation training (AOT) is a therapeutic approach used in stroke rehabilitation. Videos form the core of AOT, and knowledge of constituent parameters is essential to make the intervention robust and generalizable. Currently, there is a dearth of available information on video parameters to be used for AOT. Our purpose was to identify and describe the parameters that constitute AOT videos for stroke rehabilitation. Method Electronic databases like PubMed, CINAHL, Scopus, Web of Science, ProQuest, and Ovid SP from inception to date according to PRISMA-ScR guidelines. Title, abstract, and full-text screening were done independently by two authors, with a third author for conflict resolution. Data on video parameters like length, quality, perspective, speed, screen size and distance, sound, and control videos were extracted. Results Seventy studies were included in this review. The most-reported parameters were video length (85.71%) and perspective of view (62.85%). Movement speed (7.14%) and sound (8.57%) were the least reported. Static landscapes or geometrical patterns were found suitable as control videos. Conclusion Most video parameters except for length and perspective of view remain underreported in AOT protocols. Future studies with better descriptions of video parameters are required for comprehensive AOT interventions and result generalisation.

A model for transforming egocentric views into goal-directed behavior.

Neurons in the rat postrhinal cortex (POR) respond to the egocentric (observer-centered) bearing and distance of the boundaries, or geometric center, of an enclosed space. Understanding of the precise geometric and sensory properties of the environment that generate these signals is limited. Here we model how this signal may relate to visual perception of motion parallax along environmental boundaries. A behavioral extension of this tuning is the known 'centering response', in which animals follow a spatial gradient function based on boundary parallax to guide behavior toward the center of a corridor or enclosure. Adding an allocentric head direction signal to this representation can translate the gradient across two-dimensional space and provide a new gradient for directing behavior to any location. We propose a model for how this signal may support goal-directed navigation via projections to the dorsomedial striatum. The result is a straightforward code for navigational variables derived from visual geometric properties of the surrounding environment, which may be used to map space and transform incoming sensory information into an appropriate motor output.

Quarter-Life Crisis in Generation Z Adults

Quarter-life crisis is a stage of emotional crisis that occurs in individuals in their 20s including feelings of helplessness, isolation, self-doubt, and fear of failure. Meanwhile, generation Z or iGen refers to the generation that is in the period between 1995-2010 which can also be called iGeneration, net generation, or internet generation. Generation Z or iGen has special characteristics regarding the quarter-life crisis and shows different behavior when facing the quarter-life crisis. Early adulthood itself is a time of shifting from an egocentric view to an empathetic attitude. The purpose of this study is to reveal the description of the quarter-life crisis in early adults of Generation Z or iGen. This research is descriptive qualitative research. Interviews are used as a method of extracting data on one informant. The results of this study are the subjects experience frustration when they cannot motivate themselves, panic over pressure from the family environment and social suspicion, experience helplessness when listening to music that is listened to as if it follows their life, and the inability to multitask. in their social life. have a purpose in life because they are haunted by an uncertain future, and unhappiness because the subjects is unsure of their partner and other problems related to opposite-sex relationships to friendship.

Real-time hand ownership decision in egocentric view using kinematic approach

Real-time hand ownership decision in egocentric view using kinematic approach

Vision-Based Efficient Robotic Manipulation with a Dual-Streaming Compact Convolutional Transformer.

Learning from visual observation for efficient robotic manipulation is a hitherto significant challenge in Reinforcement Learning (RL). Although the collocation of RL policies and convolution neural network (CNN) visual encoder achieves high efficiency and success rate, the method general performance for multi-tasks is still limited to the efficacy of the encoder. Meanwhile, the increasing cost of the encoder optimization for general performance could debilitate the efficiency advantage of the original policy. Building on the attention mechanism, we design a robotic manipulation method that significantly improves the policy general performance among multitasks with the lite Transformer based visual encoder, unsupervised learning, and data augmentation. The encoder of our method could achieve the performance of the original Transformer with much less data, ensuring efficiency in the training process and intensifying the general multi-task performances. Furthermore, we experimentally demonstrate that the master view outperforms the other alternative third-person views in the general robotic manipulation tasks when combining the third-person and egocentric views to assimilate global and local visual information. After extensively experimenting with the tasks from the OpenAI Gym Fetch environment, especially in the Push task, our method succeeds in 92% versus baselines that of 65%, 78% for the CNN encoder, 81% for the ViT encoder, and with fewer training steps.

Non-Autoregressive Transformer Based Ego-Motion Independent Pedestrian Trajectory Prediction on Egocentric View

Non-Autoregressive Transformer Based Ego-Motion Independent Pedestrian Trajectory Prediction on Egocentric View

EgoFish3D: Egocentric 3D Pose Estimation From a Fisheye Camera via Self-Supervised Learning

Egocentric vision has gained increasing popularity recently, opening new avenues for human-centric applications. However, the use of the egocentric fisheye cameras allows wide angle coverage but image distortion is introduced along with strong human body self-occlusion imposing significant challenges in data processing and model reconstruction. Unlike previous work only leveraging synthetic data for model training, this paper presents a new real-world EgoCentric Human Pose (ECHP) dataset. To tackle the difficulty of collecting 3D ground truth using motion capture systems, we simultaneously collect images from a head-mounted egocentric fisheye camera as well as from two third-person-view cameras, circumventing the environmental restrictions. By using self-supervised learning under multi-view constraints, we propose a simple yet effective framework, namely EgoFish3D, for egocentric 3D pose estimation from a single image in different real-world scenarios. The proposed EgoFish3D incorporates three main modules. 1) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">The third-person-view module</i> takes two exocentric images as input and estimates the 3D pose represented in the third-person camera frame; 2) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">the egocentric module</i> predicts the 3D pose in the egocentric camera frame; and 3) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">the interactive module</i> estimates the rotation matrix between the third-person and the egocentric views. Experimental results on our ECHP dataset and existing benchmark datasets demonstrate the effectiveness of the proposed EgoFish3D, which can achieve superior performance to existing methods.

X-view: Non-egocentric Multi-View 3D Object Detector.

3D object detection algorithms for autonomous driving reason about 3D obstacles either from 3D birds-eye view or perspective view or both. Recent works attempt to improve the detection performance via mining and fusing from multiple egocentric views. Although the egocentric perspective view alleviates some weaknesses of the birds-eye view, the sectored grid partition becomes so coarse in the distance that the targets and surrounding context mix together, which makes the features less discriminative. In this paper, we generalize the research on 3D multi-view learning and propose a novel multi-view-based 3D detection method, named X-view, to overcome the drawbacks of the multi-view methods. Specifically, X-view breaks through the traditional limitation about the perspective view whose original point must be consistent with the 3D Cartesian coordinate. X-view is designed as a general paradigm that can be applied on almost any 3D detectors based on LiDAR with only little increment of running time, no matter it is voxel/grid-based or raw-point-based. We conduct experiments on KITTI [1] and NuScenes [2] datasets to demonstrate the robustness and effectiveness of our proposed X-view. The results show that X-view obtains consistent improvements when combined with mainstream state-of-the-art 3D methods.

ViewBirdiformer: Learning to Recover Ground-Plane Crowd Trajectories and Ego-Motion From a Single Ego-Centric View

We introduce a novel learning-based method for view birdification [1], the task of recovering ground-plane trajectories of pedestrians of a crowd and their observer in the same crowd just from the observed ego-centric video. View birdification becomes essential for mobile robot navigation and localization in dense crowds where the static background is hard to see and reliably track. It is challenging mainly for two reasons; i) absolute trajectories of pedestrians are entangled with the movement of the observer which needs to be decoupled from their observed relative movements in the ego-centric video, and ii) a crowd motion model describing the pedestrian movement interactions is specific to the scene yet unknown a priori. For this, we introduce a Transformer-based network referred to as ViewBirdiformer which implicitly models the crowd motion through self-attention and decomposes relative 2D movement observations onto the ground-plane trajectories of the crowd and the camera through cross-attention between views. Most important, ViewBirdiformer achieves view birdification in a single forward pass which opens the door to accurate real-time, always-on situational awareness. Extensive experimental results demonstrate that ViewBirdiformer achieves accuracy similar to or better than state-of-the-art with three orders of magnitude reduction in execution time.

Projective Bisector Mirror (PBM): Concept and Rationale.

Our world is full of cameras, whether they are installed in the environment or integrated into mobile devices such as mobile phones or head-mounted displays. Displaying external camera views in our egocentric view with a picture-in-picture approach allows us to understand their view; however, it would not allow us to correlate their viewpoint with our perceived reality. We introduce Projective Bisector Mirrors for visualizing a camera view comprehensibly in the egocentric view of an observer with the metaphor of a virtual mirror. Our concept projects the image of a capturing camera onto the bisecting plane between the capture and the observer camera. We present extensive mathematical descriptions of this novel paradigm for multi-view visualization, discuss the effects of tracking errors and provide concrete implementation for multiple exemplary use-cases.

Scene saliencies in egocentric vision and their creation by parents and infants

Across the lifespan, humans are biased to look first at what is easy to see, with a handful of well-documented visual saliences shaping our attention (e.g., Itti & Koch, 2001). These attentional biases may emerge from the contexts in which moment-tomoment attention occurs, where perceivers and their social partners actively shape bottom-up saliences, moving their bodies and objects to make targets of interest more salient. The goal of the present study was to determine the bottom-up saliences present in infant egocentric images and to provide evidence on the role that infants and their mature social partners play in highlighting targets of interest via these saliences. We examined 968 unique scenes in which an object had purposefully been placed in the infant's egocentric view, drawn from videos created by one-year-old infants wearing a head camera during toy-play with a parent. To understand which saliences mattered in these scenes, we conducted a visual search task, asking participants (n = 156) to find objects in the egocentric images. To connect this to the behaviors of perceivers, we then characterized the saliences of objects placed by infants or parents compared to objects that were otherwise present in the scenes. Our results show that body-centric properties, such as increases in the centering and visual size of the object, as well as decreases in the number of competing objects immediately surrounding it, both predicted faster search time and distinguished placed and unplaced objects. The present results suggest that the bottom-up saliences that can be readily controlled by perceivers and their social partners may most strongly impact our attention. This finding has implications for the functional role of saliences in human vision, their origin, the social structure of perceptual environments, and how the relation between bottom-up and top-down control of attention in these environments may support infant learning.

Head and Eye Egocentric Gesture Recognition for Human-Robot Interaction Using Eyewear Cameras

Non-verbal communication plays a particularly important role in a wide range of scenarios in Human-Robot Interaction (HRI). Accordingly, this work addresses the problem of human gesture recognition. In particular, we focus on head and eye gestures, and adopt an egocentric (first-person) perspective using eyewear cameras. We argue that this egocentric view may offer a number of conceptual and technical benefits over scene- or robot-centric perspectives. A motion-based recognition approach is proposed, which operates at two temporal granularities. Locally, frame-to-frame homographies are estimated with a convolutional neural network (CNN). The output of this CNN is input to a long short-term memory (LSTM) to capture longer-term temporal visual relationships, which are relevant to characterize gestures. Regarding the configuration of the network architecture, one particularly interesting finding is that using the output of an internal layer of the homography CNN increases the recognition rate with respect to using the homography matrix itself. While this work focuses on action recognition, and no robot or user study has been conducted yet, the system has been designed to meet real-time constraints. The encouraging results suggest that the proposed egocentric perspective is viable, and this proof-of-concept work provides novel and useful contributions to the exciting area of HRI.