Third-person Video Research Articles

Automatic Detection of Maintenance Scenarios for Equipment and Control Systems in Industry

The well-known methods of scene extraction on video are focused on analyzing the similarity between frames. However, they do not all analyze the composition of the image scene, which may remain the same during maintenance. Therefore, this paper proposes an algorithm for equipment maintenance scene detection based on human hand tracking. It is based on the assumption that, when servicing technological equipment, it is possible to determine the change in repair action by the position of the service engineer’s hands. Thus, certain information and the algorithm that processes these changes allow us to segment the video into actions performed during the service. We process the time series obtained by moving the hand position using spectral singular value decomposition for multivariate time series. To verify the algorithm, we performed maintenance on the control cabinet of a mining conveyor and recorded the work on a first-person video, which was processed using the developed method. As a result, we obtained some scenes corresponding to opening the control cabinet, de-energizing the unit, and checking the contacts with a multimeter buzzer test. A third-person video of motor service was similarly processed. The algorithm demonstrated the results in separate scenes of removing screws, working with a multimeter, and disconnecting and replacing motor parts.

Understanding First-Person and Third-Person Videos in Computer Vision

Due to advancements in technology and social media, a large amount of visual information is created. There is a lot of interesting research going on in Computer Vision that takes into consideration either visual information generated by first-person (egocentric) or third-person(exocentric) cameras. Video data generated by YouTubers, Surveillance cameras, and Drones which is referred to as third-person or exocentric video data. Whereas first-person or egocentric is the one which is generated by GoPro cameras and Google Glass. Exocentric view capture wide and global views whereas egocentric view capture activities an actor is involved in w.r.t. objects. These two perspectives seem to be independent yet related. In Computer Vision, these two perspectives have been studied by various domains like Activity Recognition, Object Detection, Action Recognition, and Summarization independently. Their relationship and comparison are less discussed in the literature. This paper tries to bridge this gap by presenting a systematic study of first-person and third-person videos. Further, we implemented an algorithm to classify videos as first-person/third-person with the validation accuracy of 88.4% and an F1-score of 86.10% using the Charades dataset..

Ego-Body Pose Estimation via Ego-Head Pose Estimation

Estimating 3D human motion from an ego-centric video, which records the environment viewed from the first-person perspective with a front-facing monocular camera, is critical to applications in VR/AR. However, naively learning a mapping between egocentric videos and full-body human motions is challenging for two reasons. First, modeling this complex relationship is difficult; unlike reconstruction motion from third-person videos, the human body is often out of view of an egocentric video. Second, learning this mapping requires a large-scale, diverse dataset containing paired egocentric videos and the corresponding 3D human poses. Creating such a dataset requires meticulous instrumentation for data acquisition, and unfortunately, such a dataset does not currently exist. As such, existing works have only worked on small-scale datasets with limited motion and scene diversity (yuan20183d; yuan2019ego; luo2021dynamics).

Beyond accidents: Young children’s forgiveness of third-party intentional transgressors

To maintain our cooperative relationships, it is critical that we repair these relationships when they are damaged by transgressions. Key to this repair is forgiveness. Previous research suggests that adults and children are more forgiving of remorseful transgressors than of unremorseful ones after accidental transgressions. However, little is known about whether children’s forgiveness also takes the transgressors’ intentions into account. Using a third-person video paradigm with children in the United States, Study 1 found that 6-year-olds (n = 20; 10 girls; 60% White) were more likely to negatively evaluate an intentional transgressor and give more resources to an accidental transgressor when both transgressors showed remorse, whereas 5-year-olds (n = 20; 10 girls; 80% White) showed this effect only in their resource distribution. Study 2 found that 6-year-olds (n = 18; 7 girls; 83% White) were more likely to positively evaluate and share more resources with a remorseful intentional transgressor than with an unremorseful intentional one. Thus, by school age in the United States, children’s forgiveness, at least as bystanders, begins to take into account both the transgressor’s intentions and the display of remorse.

Automated detections reveal the social information in the changing infant view.

How do postural developments affect infants' access to social information? We recorded egocentric and third-person video while infants and their caregivers (N=36, 8- to 16-month-olds, N=19 females) participated in naturalistic play sessions. We then validated the use of a neural network pose detection model to detect faces and hands in the infant view. We used this automated method to analyze our data and a prior egocentric video dataset (N=17, 12-month-olds). Infants' average posture and orientation with respect to their caregiver changed dramatically across this age range; both posture and orientation modulated access to social information. Together, these results confirm that infant's ability to move and act on the world plays a significant role in shaping the social information in their view.

Presence experienced in smartphone-based exposure: First and third person perspectives

Various formats to deliver exposure stimuli have been developed, including video-based stimuli. Virtual reality-based exposure often utilizes a first-person perspective, which is associated with greater presence (e.g., feeling integrated in a virtual world) than third-person perspective. Yet, few have compared exposure stimuli presented in first-person versus third-person perspective. Thus, this study examined presence and anxiety levels associated with exposure video perspective. Participants ( N = 18) completed a two-week video-based exposure treatment (i.e., watched one week of first-person perspective film and one week of third-person perspective film, counterbalanced). Participants reported anxiety and presence felt during the exposures. Results indicated greater presence was experienced during first-person videos than third-person ones. Anxiety levels did not differ between video perspective. An interaction between perspective and video order was found; those who watched third-person videos during week one reported more anxiety when presented first-person videos than those who watched the reverse order. Results support previous findings that more presence is reported with first-person than third-person perspective. More importantly, this study suggests an innovative way to sequence exposure stimuli so as to maximally sustain therapeutic levels of arousal over the course of treatment.

Learning representative temporal features for action recognition

In this paper, a novel video classification method is presented that aims to recognize different categories of third-person videos efficiently. Our motivation is to achieve a light model that could be trained with insufficient training data. With this intuition, the processing of the 3-dimensional video input is broken to 1D in temporal dimension on top of the 2D in spatial. The processes related to 2D spatial frames are being done by utilizing pre-trained networks with no training phase. The only step which involves training is to classify the 1D time series resulted from the description of the 2D signals. As a matter of fact, optical flow images are first calculated from consecutive frames and described by pre-trained CNN networks. Their dimension is then reduced using PCA. By stacking the description vectors beside each other, a multi-channel time series is created for each video. Each channel of the time series represents a specific feature and follows it over time. The main focus of the proposed method is to classify the obtained time series effectively. Towards this, the idea is to let the machine learn temporal features. This is done by training a multi-channel one dimensional Convolutional Neural Network (1D-CNN). The 1D-CNN learns the features along the only temporal dimension. Hence, the number of training parameters decreases significantly which would result in the trainability of the method on even smaller datasets. It is illustrated that the proposed method could reach the state-of-the-art results on two public datasets UCF11, jHMDB and competitive results on HMDB51.

Self-Regulated Learning for Egocentric Video Activity Anticipation.

Future activity anticipation is a challenging problem in egocentric vision. As a standard future activity anticipation paradigm, recursive sequence prediction suffers from the accumulation of errors. To address this problem, we propose a simple and effective Self-Regulated Learning framework, which aims to regulate the intermediate representation consecutively to produce representation that (a) emphasizes the novel information in the frame of the current time-stamp in contrast to previously observed content, and (b) reflects its correlation with previously observed frames. The former is achieved by minimizing a contrastive loss, and the latter can be achieved by a dynamic reweighing mechanism to attend to informative frames in the observed content with a similarity comparison between feature of the current frame and observed frames. The learned final video representation can be further enhanced by multi-task learning which performs joint feature learning on the target activity labels and the automatically detected action and object class tokens. SRL sharply outperforms existing state-of-the-art in most cases on two egocentric video datasets and two third-person video datasets. Its effectiveness is also verified by the experimental fact that the action and object concepts that support the activity semantics can be accurately identified.

First- And Third-Person Video Co-Analysis By Learning Spatial-Temporal Joint Attention.

Recent years have witnessed a tremendous increase of first-person videos captured by wearable devices. Such videos record information from different perspectives than the traditional third-person view, and thus show a wide range of potential usages. However, techniques for analyzing videos from different views can be fundamentally different, not to mention co-analyzing on both views to explore the shared information. In this paper, we take the challenge of cross-view video co-analysis and deliver a novel learning-based method. At the core of our method is the notion of "joint attention", indicating the shared attention regions that link the corresponding views, and eventually guide the shared representation learning across views. To this end, we propose a multi-branch deep network, which extracts cross-view joint attention and shared representation from static frames with spatial constraints, in a self-supervised and simultaneous manner. In addition, by incorporating the temporal transition model of the joint attention, we obtain spatial-temporal joint attention that can robustly capture the essential information extending through time. Our method outperforms the state-of-the-art on the standard cross-view video matching tasks on public datasets. Furthermore, we demonstrate how the learnt joint information can benefit various applications through a set of qualitative and quantitative experiments.

Learning With Media

Abstract. Digital media, such as interactive video, games, and immersive worlds, offer rich visual perspectives, often allowing one to experience events through another’s eyes. While prior research indicates that considering alternative perspectives facilitates understanding, little is known about how media-enhanced perspectives affect learning processes for higher-order concepts that require synthesis of ideas and making inferences such as reasoning about problems in science. Two experiments used digital video of a science instructional event to investigate features of visual perspective on engagement and knowledge construction. Study 1 showed that an embodied first-person viewpoint achieved using a head-mounted camera better supported learning than a traditional third-person view of the same event. In Study 2, applying a motion algorithm to both a first-person and third-person video allowed us to isolate the effects of viewpoint and camera motion. While the addition of artificial motion benefited learning for third-person viewers, only motion that is aligned with the actor’s actions and affect enhances first-person viewing. Findings are considered in terms of how certain media position learners in relation to educational content. Specifically, we argue that media features such as viewpoint and motion can be configured in ways to create “fields of potential action” that engage viewers and optimize conditions for learning.

Moving in stealth: On the tracking shot as a technique for imperceptibility

This article explores recent uses of the tracking shot in various media. Examples drawn from television, video games and video art reveal that recent audiovisual media have frequently used a particular kind of tracking shot that follows an individual through a complex environment. This article argues that this tracking shot contributes to an aesthetic of stealth, that is, a perceptual attunement to notions of imperceptibility and secrecy. The stealth tracking shot can thus be seen as one of the aesthetic principles that articulate discourses of securitisation. The argument proceeds in three steps. First, the article shows that this specific use of the tracking shot takes inspiration from third-person video games. An analysis of the stealth game Splinter Cell: Blacklist (2013) shows how the mobile ‘following camera’ allows the viewer to perceive the avatar in his or her environment. Then, in an analysis of the unbroken tracking shot in True Detective (2014), it is demonstrated how this kind of tracking shot contributes to a reformulation of the notions of law enforcement underlying crime fiction. Specifically, the tracking shot is related to the notion of stealth democracy. Finally, the article considers Hito Steyerl’s video ‘Guards’ to show that the aesthetic principles of stealth operate not only in individual media objects but more generally in public spaces and institutions such as museums and art galleries. To conclude, the article situates the stealth tracking shot in a more general consideration of the politicality of media aesthetics.

Development and Validation of a Sensor-Based Algorithm for Detecting the Visual Exploratory Actions

Wearable sensors are becoming widely used in the sport sciences to assess the performance of athletes. Advances in microelectromechanical systems technology, in particular inertial measurement units (IMUs), provide researchers and practitioners with a portable means of capturing performance in representative task scenarios. Of recent interest to sport scientists in team sports is how athletes perceive their surroundings and how visual (and other) information is used to select appropriate actions during a match. Collectively, the movements athletes make to gather information from their environments is referred to as exploratory action. An important aspect of this behavior is typically measured by notating (counting) the number of head turns from a third-person video perspective. A notational analysis is a labor-intensive task and prone to human error, especially when activity takes place over long durations. The IMUs are well suited to resolve these issues; they are highly accurate, very efficient, and have an adequate output data rate. Currently, no gold standard method exists to automatically detect head turn events from the IMUs. In the current study, a novel algorithm that utilizes data captured from a head-mounted IMU to count the number of head turns performed by an athlete during a controlled experimental task is presented. Results demonstrate that the presented algorithm is an appropriate and efficient method for assessing the number of head turns as a measure of exploratory actions.

The role of perspective in event segmentation

People divide their ongoing experience into meaningful events. This process, event segmentation, is strongly associated with visual input: when visual features change, people are more likely to segment. However, the nature of this relationship is unclear. Segmentation could be bound to specific visual features, such as actor posture. Or, it could be based on changes in the activity that are correlated with visual features. This study distinguished between these two possibilities by examining whether segmentation varies across first- and third-person perspectives. In two experiments, observers identified meaningful events in videos of actors performing everyday activities, such as eating breakfast or doing laundry. Each activity was simultaneously recorded from a first-person perspective and a third-person perspective. These videos presented identical activities but differed in their visual features. If segmentation is tightly bound to visual features then observers should identify different events in first- and third-person videos. In addition, the relationship between segmentation and visual features should remain unchanged. Neither prediction was supported. Though participants sometimes identified more events in first-person videos, the events they identified were mostly indistinguishable from those identified for third-person videos. In addition, the relationship between the video’s visual features and segmentation changed across perspectives, further demonstrating a partial dissociation between segmentation and visual input. Event segmentation appears to be robust to large variations in sensory information as long as the content remains the same. Segmentation mechanisms appear to flexibly use sensory information to identify the structure of the underlying activity.

ReHabgame: A non-immersive virtual reality rehabilitation system with applications in neuroscience

This paper proposes the use of a non-immersive virtual reality rehabilitation system “ReHabgame” developed using Microsoft Kinect™ and the Thalmic™ Labs Myo gesture control armband. The ReHabgame was developed based on two third-person video games that provide a feasible possibility of assessing postural control and functional reach tests. It accurately quantifies specific postural control mechanisms including timed standing balance, functional reach tests using real-time anatomical landmark orientation, joint velocity, and acceleration while end trajectories were calculated using an inverse kinematics algorithm. The game was designed to help patients with neurological impairment to be subjected to physiotherapy activity and practice postures of daily activities. The subjective experience of the ReHabgame was studied through the development of an Engagement Questionnaire (EQ) for qualitative, quantitative and Rasch model.The Monte-Carlo Tree Search (MCTS) and Random object (ROG) generator algorithms were used to adapt the physical and gameplay intensity in the ReHabgame based on the Motor Assessment Scale (MAS) and Hierarchical Scoring System (HSS). Rasch analysis was conducted to assess the psychometric characteristics of the ReHabgame and to identify if these are any misfitting items in the game.Rasch rating scale model (RSM) was used to assess the engagement of players in the ReHabgame and evaluate the effectiveness and attractiveness of the game. The results showed that the scales assessing the rehabilitation process met Rasch expectations of reliability, and unidimensionality. Infit and outfit mean squares values are in the range of (0.68–1.52) for all considered 16 items. The Root Mean Square Residual (RMSR) and the person separation reliability were acceptable. The item/person map showed that the persons and items were clustered symmetrically.

Видеомоделирование от первого лица как способ обучения имитации детей с аутизмом

The article discusses the problem of the basic factors of ASD: the deficit in the Theory of Mind (ToM) as the ability to represent Self-Other relationship, or violation of the ability to simulate, as the ability to transfer the perspective of Other’s action into the prospect of their own actions. It is hypothesized that the first-person video surveillance technique (with the instructor’s and the student's hands visible) will be more effective in teaching imitation of children with ASD than the third-person video surveillance method (where the instructor and the student are seen). The study involved 28 children attending classes with psychologists and speech pathologists (ages: 4, 10 up to 7, 4 years; 24 children were diagnosed with ASD, 4 – atypical autism). At the first stage, the level of motor simulation skills (ABLLS-R test) was evaluated-3 times with an interval of 1 week. On the second stage, three groups (9 people) were equalized at the level of development of imitation. One group was trained "third-person", the second – "first-person", the third – control was trained according to the standard program. All studied the same time. For each subject conducted 4 classes (2 times a week). At the third stage, the level of motor skills measured 3 times with an interval of 1 week. The processing took into account the parameters of the mean, median, mode and standard deviation. The results of the study showed significant effectiveness of third-person training. The shift of the average value in "third-person training" and "first-person training" was 3 and 8 units respectively.

Why Can I See My Avatar? Embodied Visual Engagement in the Third-Person Video Game

This essay seeks to answer two questions raised by the success of video games where the player looks at the character she is playing rather than seeming to inhabit the same coordinates as the character within the game space. First, why is the experience of playing these games not innately inferior to that of playing games with a first-person point of view, given that the sense of being a character sensing and acting inside the game space could be expected to be much stronger when the character’s body seems to be one’s own rather than a separate entity in the game space? And second, if the first-person point of view is so “immersive” and provides such a sense of being “inside” the representational space as is sometimes claimed, why has it never been so prominent in other audiovisual entertainment media such as film and television?

Repetition and Ritual Logic in Video Games

By moving away from a model of ritual that focuses on magic and fantasy worlds, this article seeks to broaden the discussion of ritual actions, performances, and objects in first- and third-person video games. Ritual will be understood through the idea of a “ritual logic” that enables the wider associations of ritual in the virtual as opposed to the real world to be analyzed, and through the key element of repetition in game play. In part derived from the intertextuality of video game genres and associated popular culture artifacts such as films and novels, ritual logic contributes to the players’ knowledge and understanding of what ritual is and what ritual does in the game, and how ritual can be used to progress its narrative and play trajectories.

Observation of Action and Autonomic Nervous System Responses

Observing somebody performing an action has been shown to elicit neuronal activity in the premotor cortex. This paper investigated physiological effect of observing an effortful action at the peripheral level. As Autonomic Nervous System responses reflect central nervous system processes such as movement planning and programming, it was expected that observing an action would elicit a pattern of ANS responses matching those recorded during actual movement. 12 male subjects, ages 23 to 28 years (M = 25.5, SD = 1.9), were selected as they were experienced in weight lifting. They were asked to observe a squat movement followed by returning to the upright position under 3 different conditions: (i) observation of actual movement performed by somebody else, (ii) observation of a video of the subject himself (first-person video), and (iii) observation of a video of somebody else performing the same movement (third-person video). Moreover, each movement was observed when performed at 50% and 90% of each participant's personal best mark (% of the highest weight which could be lifted). Three ANS parameters were continuously recorded: skin resistance, temperature and heart rate. ANS responses varied as a function of movement intensity: autonomic responses recorded during movement observation at 90% were significantly higher and longer than those recorded during movement observation at 50%. Thus, autonomic responses were linked to the amount of observed effort. Conversely, no difference was found among the three conditions of observation. ANS responses from observation of actual movement were shown to resemble those recorded under the two conditions of video observation.