First-person View Videos Research Articles

Video saliency prediction for First-Person View UAV videos: Dataset and benchmark

Visual saliency prediction plays a crucial role in Unmanned Aerial Vehicle (UAV) video analysis tasks. In this paper, an eye-tracking dataset of the immersive viewing of videos captured from a First-Person View (FPV) of UAVs is developed, which consists of 200 video clips captured by DJI FPV drones, with a resolution of 4K QHD. The videos cover six different genres and fourteen unique scenes. To study human visual attention in watching FPV videos, fixation points are recorded using an eye tracker integrated into a VR headset. Based on the dataset, a simple yet effective FPV UAV video Saliency prediction model (FUAVSal) is proposed as a baseline, considering spatial–temporal feature, camera motion information and FPV prior. To establish benchmarks for saliency prediction in immersive FPV UAV video viewing, sixteen computational models are evaluated on this dataset. Detailed quantitative and qualitative comparisons are provided. The developed dataset and benchmarks aim to facilitate research on visual saliency prediction for First-Person View UAV videos.

Analyzing students' attention by gaze tracking and object detection in classroom teaching

PurposeAttention is one of the most important factors to affect the academic performance of students. Effectively analyzing students' attention in class can promote teachers' precise teaching and students' personalized learning. To intelligently analyze the students' attention in classroom from the first-person perspective, this paper proposes a fusion model based on gaze tracking and object detection. In particular, the proposed attention analysis model does not depend on any smart equipment.Design/methodology/approachGiven a first-person view video of students' learning, the authors first estimate the gazing point by using the deep space–time neural network. Second, single shot multi-box detector and fast segmentation convolutional neural network are comparatively adopted to accurately detect the objects in the video. Third, they predict the gazing objects by combining the results of gazing point estimation and object detection. Finally, the personalized attention of students is analyzed based on the predicted gazing objects and the measurable eye movement criteria.FindingsA large number of experiments are carried out on a public database and a new dataset that is built in a real classroom. The experimental results show that the proposed model not only can accurately track the students' gazing trajectory and effectively analyze the fluctuation of attention of the individual student and all students but also provide a valuable reference to evaluate the process of learning of students.Originality/valueThe contributions of this paper can be summarized as follows. The analysis of students' attention plays an important role in improving teaching quality and student achievement. However, there is little research on how to automatically and intelligently analyze students' attention. To alleviate this problem, this paper focuses on analyzing students' attention by gaze tracking and object detection in classroom teaching, which is significant for practical application in the field of education. The authors proposed an effectively intelligent fusion model based on the deep neural network, which mainly includes the gazing point module and the object detection module, to analyze students' attention in classroom teaching instead of relying on any smart wearable device. They introduce the attention mechanism into the gazing point module to improve the performance of gazing point detection and perform some comparison experiments on the public dataset to prove that the gazing point module can achieve better performance. They associate the eye movement criteria with visual gaze to get quantifiable objective data for students' attention analysis, which can provide a valuable basis to evaluate the learning process of students, provide useful learning information of students for both parents and teachers and support the development of individualized teaching. They built a new database that contains the first-person view videos of 11 subjects in a real classroom and employ it to evaluate the effectiveness and feasibility of the proposed model.

Spatio-temporal categorization for first-person-view videos using a convolutional variational autoencoder and Gaussian processes.

In this study, HcVGH, a method that learns spatio-temporal categories by segmenting first-person-view (FPV) videos captured by mobile robots, is proposed. Humans perceive continuous high-dimensional information by dividing and categorizing it into significant segments. This unsupervised segmentation capability is considered important for mobile robots to learn spatial knowledge. The proposed HcVGH combines a convolutional variational autoencoder (cVAE) with HVGH, a past method, which follows the hierarchical Dirichlet process-variational autoencoder-Gaussian process-hidden semi-Markov model comprising deep generative and statistical models. In the experiment, FPV videos of an agent were used in a simulated maze environment. FPV videos contain spatial information, and spatial knowledge can be learned by segmenting them. Using the FPV-video dataset, the segmentation performance of the proposed model was compared with previous models: HVGH and hierarchical recurrent state space model. The average segmentation F-measure achieved by HcVGH was 0.77; therefore, HcVGH outperformed the baseline methods. Furthermore, the experimental results showed that the parameters that represent the movability of the maze environment can be learned.

Just showing is not enough: First-person-view-videos as a feedback tool in resuscitation simulation

Theory and evidence suggest beneficial effects of integrating first-person-view (FPV) videos as a feedback method in resuscitation simulation training. However, systematic research on this instructional approach is scarce. We aimed to determine the feasibility of supplementing feedback with FPV videos for simulation-based learning and its effect on cardiopulmonary resuscitation (CPR) performance. In our randomized controlled trial conducted in a simulation-based course on CPR, each participant’s FPV video was recorded using a head-mounted camera and videos were shown before post-simulation debriefing. Feasibility was tested in 27 simulated CPR scenarios randomized to FPV feedback combined with instructor-led debriefing (intervention; 14 scenarios) versus instructor-led debriefing only (control; 13 scenarios). In terms of CPR (effective compression ratio [ECR]) and team performance (Team Emergency Assessment Measure [TEAM]), improvements were observed for ECR (pre-test: 0.13 ± 0.11 vs post-test 0.19 ± 0.11) and TEAM score (pre-test: 16.6 ± 7.2 vs post-test: 25.4 ± 9.2). However, improvements did not differ significantly between intervention and control group. Course participants deemed the feedback method feasible and reported positive instructional effects. In order to make use of the potential of this promising feedback tool, we conclude that it is not sufficient to show the videos without didactical assistance.

Textile Antenna for First-Person View Goggles

Unmanned Aerial Vehicles, also known as drones, are vehicles controlled remotely. First-Person View (FPV) technology allows these vehicles to have greater versatility and be more easily piloted. The antennas currently used for video transmission are bulky and uncomfortable, so there is a need for this volume to be reduced to increase portability, being this the focus of the paper. The antenna was developed in textile materials to reduce its size and weight, as these materials allow for a better comfort and a reduced footprint. This paper presents a microstrip array adapted for the frequencies used in FPV video transmission. The designed textile antenna was compared with a commercially available rigid antenna used for this transmission. The comparison shows promising results for the antenna made of textile materials. The main goal of increasing the comfort and portability of the FPV goggles was achieved.

FPV Video Adaptation for UAV Collision Avoidance

First person view (FPV) technology for unmanned aerial vehicles (UAVs) provides an immersive experience for pilots and enables various personal and commercial applications such as aerial photography, drone racing, search and rescue operations, agricultural surveillance, and structural inspection. While real time video streaming from a UAV and vision-based collision avoidance strategies have been studied in literature as separate topics, in this paper we tackle collision avoidance in FPV scenarios, taking into account network delays and real time video parameters. We present a theoretical model for obstacle collisions that considers the current communication channel conditions, the real time video parameters, and the UAV’s position relative to the closest obstacle. A video adaptation algorithm is then designed, using this metric, to tune the FPV video resolution, number of re-transmission attempts, and the modulation scheme to maximize the probability of avoiding collisions. This algorithm also takes into account specific latency constraints of the application. This video algorithm was evaluated in various scenarios and its ability to respond to both distances to the obstacle as well as the communication channel conditions was demonstrated. It was found that, for the considered scenarios, the performance of the proposed adaptive algorithm was, on an average, 58.63% higher than the closest non-adaptive one in terms of maximizing the probability of avoiding collision. Such collision avoidance strategies could be used to make UAV FPV applications safer and more reliable.

Extraction of Hand Motions Using First-Person View Video for Teaching Motions to Robots

Extraction of Hand Motions Using First-Person View Video for Teaching Motions to Robots

Capturing Subjective First-Person View Shots with Drones for Automated Cinematography

We propose an approach to capture subjective first-person view (FPV) videos by drones for automated cinematography. FPV shots are intentionally not smooth to increase the level of immersion for the audience, and are usually captured by a walking camera operator holding traditional camera equipment. Our goal is to automatically control a drone in such a way that it imitates the motion dynamics of a walking camera operator, and, in turn, capture FPV videos. For this, given a user-defined camera path, orientation, and velocity, we first present a method to automatically generate the operator’s motion pattern and the associated motion of the camera, considering the damping mechanism of the camera equipment. Second, we propose a general computational approach that generates the drone commands to imitate the desired motion pattern. We express this task as a constrained optimization problem, where we aim to fulfill high-level user-defined goals, while imitating the dynamics of the walking camera operator and taking the drone’s physical constraints into account. Our approach is fully automatic, runs in real time, and is interactive, which provides artistic freedom in designing shots. It does not require a motion capture system, and works both indoors and outdoors. The validity of our approach has been confirmed via quantitative and qualitative evaluations.

Evaluating healthcare workers’ hand hygiene performance using first-person view video observation in a standardized patient-care scenario

Monitoring healthcare workers' (HCWs) hand hygiene (HH) performance is recommended for improving compliance. Observer biases challenge data validity, thus supplemental approaches such as video observation are needed to complement monitoring. We investigate first-person view (FPV) video observation during simulated standardized patient care handling a catheter in a study with 71 HCWs. HH performance was evaluated for (1) all HH opportunities and (2) a subset of opportunities required in an ideal work sequence, hereafter core opportunities. HCWs' acceptance of FPV video observation and usability judgments were assessed. Compliance level for core HH opportunities (M = 43.5%) was significantly higher than compliance considering all opportunities (M = 30.4%, t(70) = 8.493, P < .001). Reducing HH opportunities to core opportunities would significantly increase compliance levels from the observed average of 30.4% to 44.9% (t(70) = 12.822, P < .001). Overall, both usability ratings and acceptance of the body camera were promising. FPV video observation in simulated standardized patient care provides new instruments to evaluate HH performance beyond mere compliance rates. Our results emphasize the role of optimizing workflow in order to improve HCW's HH compliance. FPV video observation in a standardized patient care simulation is feasible and offers information for HH interventions that target actual deficiencies.

The Smart Black Box: A Value-Driven High-Bandwidth Automotive Event Data Recorder

Autonomous vehicles require reliable and resilient sensor suites and ongoing validation through fleet-wide data collection. This paper proposes a Smart Black Box (SBB) to augment traditional low-bandwidth data logging with value-driven high-bandwidth data capture. The SBB caches short-term histories of data as buffers through a deterministic Mealy machine based on data value and similarity. Compression quality for each frame is determined by optimizing the trade-off between value and storage cost. With finite storage, prioritized data recording discards low-value buffers to make room for new data. This paper formulates SBB compression decision making as a constrained multi-objective optimization problem with novel value metrics and filtering. The SBB has been evaluated on a traffic simulator which generates trajectories containing events of interest (EOIs) and corresponding first-person view videos. SBB compression efficiency is assessed by comparing storage requirements with different compression quality levels and event capture ratios. Performance is evaluated by comparing results with a traditional first-in-first-out (FIFO) recording scheme. Deep learning performance on images recorded at different compression levels is evaluated to illustrate the reproducibility of SBB recorded data.

Sign language recognition in complex background scene based on adaptive skin colour modelling and support vector machine

With the advances of wearable cameras, the user can record the first-person view videos for gesture recognition or even sign language recognition to help the deaf or hard of hearing people communicate with others. In this paper, we propose a purely vision-based sign language recognition system which can be used in complex background scene. We design an adaptive skin colour modelling method for hand segmentation so that the hand contour can be derived more accurately even when different users use our system in various light conditions. Four kinds of feature descriptors are integrated to describe the contours and the salient points of hand gestures, and support vector machine (SVM) is applied to classify hand gestures. Our recognition method is evaluated by two datasets: 1) the CSL dataset collected by ourselves in which images were captured in three different environments including complex background; 2) the public ASL dataset, in which images of the same gesture were captured in different lighting conditions. The proposed recognition method achieves acceptable accuracy rates of 100.0% and 94.0% for the CSL and ASL datasets, respectively.

Sign language recognition in complex background scene based on adaptive skin colour modelling and support vector machine

With the advances of wearable cameras, the user can record the first-person view videos for gesture recognition or even sign language recognition to help the deaf or hard of hearing people communicate with others. In this paper, we propose a purely vision-based sign language recognition system which can be used in complex background scene. We design an adaptive skin colour modelling method for hand segmentation so that the hand contour can be derived more accurately even when different users use our system in various light conditions. Four kinds of feature descriptors are integrated to describe the contours and the salient points of hand gestures, and support vector machine (SVM) is applied to classify hand gestures. Our recognition method is evaluated by two datasets: 1) the CSL dataset collected by ourselves in which images were captured in three different environments including complex background; 2) the public ASL dataset, in which images of the same gesture were captured in different lighting conditions. The proposed recognition method achieves acceptable accuracy rates of 100.0% and 94.0% for the CSL and ASL datasets, respectively.

Feasibility Study for Pose Estimation of Small UAS in Known 3D Environment Using Geometric Hashing

A novel self-localization method for small Unmanned Aerial Systems (UAS) is presented. The algorithm automatically establishes correspondence between the First-Person View (FPV) video streamed from a UAS flying in an urban environment and its 3D building model. The resulting camera pose provides a precise navigation solution in the densely structured environment. Initially, Vertical Line Features are extracted from the FPV video frames, as the forward and downward-looking camera is kept stabilized through a gimbal camera mount. Geometric hashing is then used to match the extracted image features with a database of Vertical Line Features extracted from synthetic images of the three dimensional (3D) building models. The exterior orientation parameters of the FPV video frames for localizing the UAS frames are determined by photogrammetric bundle adjustment. The results demonstrate that sub-meter accuracies in the UAS's X, Y, Z positional coordinates are achievable from flying 40 m above ground.

Personal driving diary: Automated recognition of driving events from first-person videos

In this paper, we introduce the concept of personal driving diary. A personal driving diary is a multimedia archive of a person’s daily driving experience, describing important driving events of the user with annotated videos. This paper presents an automated system that constructs such multimedia diary by analyzing videos obtained from a vehicle-mounted camera. The proposed system recognizes important interactions between the driving vehicle and the other actors in videos (e.g., accident, overtaking, etc.), and labels them together with its contextual knowledge on the vehicle (e.g., mean velocity) to construct an event log. A decision tree based activity recognizer is designed, detecting driving events of vehicles and pedestrians from the first-person view videos by analyzing their trajectories and spatio-temporal relationships. The constructed diary enables efficient searching and event-based browsing of video clips, which helps the users when retrieving videos of dangerous situations. Our experiment confirms that the proposed system reliably generates driving diaries by annotating the vehicle events learned from training examples.

Adaptive eye gaze patterns in interactions with human and artificial agents

Efficient collaborations between interacting agents, be they humans, virtual or embodied agents, require mutual recognition of the goal, appropriate sequencing and coordination of each agent's behavior with others, and making predictions from and about the likely behavior of others. Moment-by-moment eye gaze plays an important role in such interaction and collaboration. In light of this, we used a novel experimental paradigm to systematically investigate gaze patterns in both human-human and human-agent interactions. Participants in the study were asked to interact with either another human or an embodied agent in a joint attention task. Fine-grained multimodal behavioral data were recorded including eye movement data, speech, first-person view video, which were then analyzed to discover various behavioral patterns. Those patterns show that human participants are highly sensitive to momentary multimodal behaviors generated by the social partner (either another human or an artificial agent) and they rapidly adapt their gaze behaviors accordingly. Our results from this data-driven approach provide new findings for understanding micro-behaviors in human-human communication which will be critical for the design of artificial agents that can generate human-like gaze behaviors and engage in multimodal interactions with humans.