Monocular Video Sequences Research Articles

A computer vision approach to monitor activity in commercial broiler chickens using trajectory-based clustering analysis

To monitor changes in broiler behaviour related to animal health and welfare, farmers typically observe their flocks using manual observation. However, due to the labour intensive and continuous aspect of this task, the analysis of broiler behaviour could be automated using camera technology. This paper proposes a proof-of-concept camera surveillance system based on the automated detection and tracking of broilers to monitor activity bouts using unsupervised 2D trajectory clustering. Firstly, a convolutional neural network-based detector was trained and tested on our labelled dataset which resulted in a precision, recall and f score of 0.98, 0.90 and 0.94, respectively. Using a tracking-by-detection approach, the proposed system was able to track chickens across video frames with a multi-object tracking accuracy of 74.7%. A component-based feature saliency Gaussian mixture model (CFSGMM) was subsequently generated and applied to objectively cluster the trajectories based on their spatiotemporal information. Nineteen features were extracted from the trajectories, representing both static and dynamic characteristics of broiler movement, and three activity classes were identified: ‘least active/resting,’ ‘active’ and ‘highly-active.’ The proposed method was validated on one-minute monocular video sequences. CFSGMM was applied to cluster 2D trajectories relating to broiler activity bouts within the commercial rearing environment with an agreement ranging from 6.0 to 99.7% when compared to human observation. We demonstrate the potential of the computer vision system to monitor overt, short-term changes in broiler activity associated with on-farm events and discuss the opportunities of leveraging the technology to monitor longer term changes in welfare state. It is anticipated that further development of the detection and tracking systems will improve the performance of the trajectory clustering method.

PVP: Personalized Video Prior for Editable Dynamic Portraits using StyleGAN

AbstractPortrait synthesis creates realistic digital avatars which enable users to interact with others in a compelling way. Recent advances in StyleGAN and its extensions have shown promising results in synthesizing photorealistic and accurate reconstruction of human faces. However, previous methods often focus on frontal face synthesis and most methods are not able to handle large head rotations due to the training data distribution of StyleGAN. In this work, our goal is to take as input a monocular video of a face, and create an editable dynamic portrait able to handle extreme head poses. The user can create novel viewpoints, edit the appearance, and animate the face. Our method utilizes pivotal tuning inversion (PTI) to learn a personalized video prior from a monocular video sequence. Then we can input pose and expression coefficients to MLPs and manipulate the latent vectors to synthesize different viewpoints and expressions of the subject. We also propose novel loss functions to further disentangle pose and expression in the latent space. Our algorithm shows much better performance over previous approaches on monocular video datasets, and it is also capable of running in real‐time at 54 FPS on an RTX 3080.

Learning metric volume estimation of fruits and vegetables from short monocular video sequences

We present a novel approach for extracting metric volume information of fruits and vegetables from short monocular video sequences and associated inertial data recorded with a hand-held smartphone. Estimated segmentation masks from a pre-trained object detector are fused with the predicted change in relative pose obtained from the inertial data to predict the class and volume of the objects of interest. Our approach works with simple RGB video frames and inertial data which are readily available from modern smartphones. It does not require reference objects of known size in the video frames. Using a balanced validation dataset, we achieve a classification accuracy of 95% and a mean absolute percentage error for the volume prediction of 16% on untrained objects, which is comparable to state-of-the-art results requiring more elaborated data recording setups. A very accurate estimation of the model uncertainty is achieved through ensembling and the use of Gaussian negative log-likelihood loss. The dataset used in our experiments including ground-truth volume information is available at https://sst.aau.at/cns/datasets.

Monocular weakly supervised depth and pose estimation method based on multi-information fusion

The depth and pose information are the basic issues in the field of robotics, autonomous driving, and virtual reality, and are also the focus and difficult issues of computer vision research. The supervised monocular depth and pose estimation learning are not feasible in environments where labeled data is not abundant. Self-supervised monocular video methods can learn effectively only by applying photometric constraints without expensive ground true depth label constraints, which results in an inefficient training process and suboptimal estimation accuracy. To solve these problems, a monocular weakly supervised depth and pose estimation method based on multi-information fusion is proposed in this paper. First, we design a high-precision stereo matching method to generate a depth and pose data as the "Ground Truth" labels to solve the problem that the ground truth labels are difficult to obtain. Then, we construct a multi-information fusion network model based on the "Ground truth" labels, video sequence, and IMU information to improve the estimation accuracy. Finally, we design the loss function of supervised cues based on "Ground Truth" labels cues and self-supervised cues to optimize our model. In the testing phase, the network model can separately output high-precision depth and pose data from a monocular video sequence. The resulting model outperforms mainstream monocular depth and poses estimation methods as well as the partial stereo matching method in the challenging KITTI dataset by only using a small number of real training data(200 pairs).

Mean Shift Fusion Color Histogram Algorithm for Nonrigid Complex Target Tracking in Sports Video

We analyze and study the tracking of nonrigid complex targets of sports video based on mean shift fusion color histogram algorithm. A simple and controllable 3D template generation method based on monocular video sequences is constructed, which is used as a preprocessing stage of dynamic target 3D reconstruction algorithm to achieve the construction of templates for a variety of complex objects, such as human faces and human hands, broadening the use of the reconstruction method. This stage requires video sequences of rigid moving target objects or sets of target images taken from different angles as input. First, the standard rigid body method of Visuals is used to obtain the external camera parameters of the sequence frames as well as the sparse feature point reconstruction data, and the algorithm has high accuracy and robustness. Then, a dense depth map is computed for each input image frame by the Multi-View Stereo algorithm. The depth reconstruction with a too high resolution not only increases the processing time significantly but also generates more noise, so the resolution of the depth map is controlled by parameters. The multiple hypothesis target tracking algorithms are used to track multiple targets, while the chunking feature is used to solve the problem of mutual occlusion and adhesion between targets. After finishing the matching, the target and background models are updated online separately to ensure the validity of the target and background models. Our results of nonrigid complex target tracking by mean shift fusion color histogram algorithm for sports video improve the accuracy by about 8% compared to other studies. The proposed tracking method based on the mean shift algorithm and color histogram algorithm can not only estimate the position of the target effectively but also depict the shape of the target well, which solves the problem that the nonrigid targets in sports video have complicated shapes and are not easy to track. An example is given to demonstrate the effectiveness and adaptiveness of the applied method.

Self-supervised monocular depth estimation with direct methods

Depth estimation is crucial to understanding the geometry of a scene in robotics and computer vision. Traditionally, depth estimators can be trained with various forms of self-supervised stereo data or supervised ground-truth data. In comparison to the methods that utilize stereo depth perception or ground-truth data from laser scans, determining depth relation using an unlabeled monocular camera proves considerably more challenging. Recent work has shown that CNN-based depth estimators can be learned using unlabeled monocular video. Without needing the stereo data or ground-truth depth data, learning with monocular self-supervised strategies can utilize much larger and more varied image datasets. Inspired by recent advances in depth estimation, in this paper, we propose a novel objective that replaces the use of explicit ground-truth depth or binocular stereo depth with unlabeled monocular video sequence data. No assumptions about scene geometry or pre-trained information are used in the proposed architecture. To enable a better pose prediction, we propose the use of an improved differentiable direct visual odometry (DDVO), which is fused with an appearance-matching loss. The auto-masking approach is introduced in the DDVO depth predictor to filter out the low-texture area or occlusion area, which can easily reduce matching error, from one frame to the subsequent frame in the monocular sequence. Additionally, we introduce a self-supervised loss function to fuse the auto-masking segment and the depth-prediction segment accordingly. Our method produces state-of-the-art results for monocular depth estimation on the KITTI driving dataset, even outperforming some supervised methods that have been trained with ground-truth depth.

MiniNet: An extremely lightweight convolutional neural network for real-time unsupervised monocular depth estimation

Predicting depth from a single image is an attractive research topic since it provides one more dimension of information to enable machines to better perceive the world. Recently, deep learning has emerged as an effective approach to monocular depth estimation. As obtaining labeled data is costly, there is a recent trend to move from supervised learning to unsupervised learning to obtain monocular depth. However, most unsupervised learning methods capable of achieving high depth prediction accuracy will require a deep network architecture which will be too heavy and complex to run on embedded devices with limited storage and memory spaces. To address this issue, we propose a new powerful network with a recurrent module to achieve the capability of a deep network while at the same time maintaining an extremely lightweight size for real-time high performance unsupervised monocular depth prediction from video sequences. Besides, a novel efficient upsample block is proposed to fuse the features from the associated encoder layer and recover the spatial size of features with the small number of model parameters. We validate the effectiveness of our approach via extensive experiments on the KITTI dataset. Our new model can run at a speed of about 110 frames per second (fps) on a single GPU, 37 fps on a single CPU, and 2 fps on a Raspberry Pi 3. Moreover, it achieves higher depth accuracy with nearly 33 times fewer model parameters than state-of-the-art models. To the best of our knowledge, this work is the first extremely lightweight neural network trained on monocular video sequences for real-time unsupervised monocular depth estimation, which opens up the possibility of implementing deep learning-based real-time unsupervised monocular depth prediction on low-cost embedded devices.

Depth estimation for a road scene using a monocular image sequence based on fully convolutional neural network

An advanced driving assistant system is one of the most popular topics nowadays, and depth estimation is an important cue for advanced driving assistant system. Depth prediction is a key problem in understanding the geometry of a road scene for advanced driving assistant system. In comparison to other depth estimation methods using stereo depth perception, determining depth relation using a monocular camera is considerably challenging. In this article, a fully convolutional neural network with skip connection based on a monocular video sequence is proposed. With the integration framework that combines skip connection, fully convolutional network and the consistency between consecutive frames of the input sequence, high-resolution depth maps are obtained with lightweight network training and fewer computations. The proposed method models depth estimation as a regression problem and trains the proposed network using a scale invariance optimization based on L2 loss function, which measures the relationships between points in the consecutive frames. The proposed method can be used for depth estimation of a road scene without the need for any extra information or geometric priors. Experiments on road scene data sets demonstrate that the proposed approach outperforms previous methods for monocular depth estimation in dynamic scenes. Compared with the currently proposed method, our method has achieved good results when using the Eigen split evaluation method. The obvious prominent one is that the linear root mean squared error result is 3.462 and the δ < 1.25 result is 0.892.

Cycle-SfM: Joint self-supervised learning of depth and camera motion from monocular image sequences.

Understanding 3D scene geometry is a fundamental research topic in computer vision, including various subproblems, such as depth prediction, visual odometry, optical flow, etc. With the advent of artificial intelligence methods like deep learning, many approaches have emerged to deal with such problems in an end-to-end manner. These pipelines take the 3D understanding task as a nonlinear optimization problem, with the purpose of minimizing the cost function of the whole framework. Here, we present a self-supervised framework for jointly learning the monocular depth and camera's ego-motion from unlabeled, unstructured, and monocular video sequences. We propose a forward-backward consistency constraint on view reconstruction to capture temporal relations across adjacent frames, whose purpose is to explore and make full use of the bidirectional projection information. A simple and practicable improvement on the design of cost function is proposed to enhance the estimated accuracy. Due to the fact that our improvement is a lightweight and general module, it can be integrated into any self-supervised architectures seamlessly, and more accurate results can be obtained. The evaluation on the KITTI dataset demonstrates that our approach is highly efficient and performs better than the existing works in pose estimation, while the results in depth estimation perform comparably with the existing ones.

Temporally Consistent Depth Prediction With Flow-Guided Memory Units

Predicting depth from a monocular video sequence is an important task for autonomous driving. Although it has advanced considerably in the past few years, recent methods based on convolutional neural networks (CNNs) discard temporal coherence in the video sequence and estimate depth independently for each frame, which often leads to undesired inconsistent results over time. To address this problem, we propose to memorize temporal consistency in the video sequence, and leverage it for the task of depth prediction. To this end, we introduce a two-stream CNN with a flow-guided memory module, where each stream encodes visual and temporal features, respectively. The memory module, implemented using convolutional gated recurrent units (ConvGRUs), inputs visual and temporal features sequentially together with optical flow tailored to our task. It memorizes trajectories of individual features selectively and propagates spatial information over time, enforcing a long-term temporal consistency to prediction results. We evaluate our method on the KITTI benchmark dataset in terms of depth prediction accuracy, temporal consistency and runtime, and achieve a new state of the art. We also provide an extensive experimental analysis, clearly demonstrating the effectiveness of our approach to memorizing temporal consistency for depth prediction.

MuDeepNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose Using Multi-view Consistency Loss

We take formulate structure from motion as a learning problem, and propose an end-to-end learning framework to calculate the image depth, optical flow, and the camera motion. This framework is composed of multiple encoder-decoder networks. The key part of the network structure is the FlowNet, which can improve the accuracy of the estimated camera ego-motion and depth. As with recent studies, we use an end-to-end learning approach with multi-view synthesis as a variety of supervision, and proposes multi-view consistency losses to constrain both depth and camera ego-motion, requiring only monocular video sequences for training. Compared to the recently popular depth-estimation-networks using a single image, our network learns to use motion parallax correction depth. Although MuDeepNet training requires the use of two adjacent frames to obtain motion parallax, it is tested by using a single image. Thus, MuDeepNet is a monocular system. The experiments on KITTI dataset show our MuDeepNet outperforms other methods.

RETRACTED ARTICLE: Analysis of moving human body detection and recovery aided training in the background of multimedia technology

Human detection and motion recovery are the hotspots in the field of artificial intelligence and computer vision. They are widely used in vehicle-assisted systems, intelligent monitoring systems, human-computer interaction, and sports training. The human body detector based on the overall feature is to calculate the feature distribution in the entire rectangular detection window, and requires most of the information of the entire human body, so the performance of this method will be significantly reduced when encountering partial occlusion or complex posture. Firstly, this paper proposes multiple parts to express the whole human body, and divides the target into several separate parts such as head, trunk and limbs, and divides and treats the individual training part classifier for the human body target of complex human posture and partial occlusion. Secondly, this paper gives a kind of a method of restoring a three-dimensional motion posture of a human body from a monocular video sequence. The experimental results show that the method can be applied to human detection with multi-pose and partial occlusion in complex background, which provides an effective technical means for human detection in complex background. At the same time, the method does not need to mark, and the recovered relative motion posture of the human body is simple and effective, the optimization result is robust, and the recovery result can satisfy the human motion posture analysis with low precision requirement.

An Image-Constrained Particle Filter for 3D Human Motion Tracking

Tracking 3D human motion from monocular video sequences has aroused great interest in recent years. Among these human motion tracking methods, the particle filter is considered as an effective approach. However, the current approaches based on particle filter still have some limitation such as many particles are obviously not consistent with the observed image due to they are independent of the image information. In this paper, we present an image-constrained particle filter approach to track 3D human motion from monocular video clips with the assistance of a pre-captured motion library. We propose two novel particle filtering criteria and design a hierarchical likelihood function. The top layer of the function consists of the particle filtering criteria, and the bottom layer consists of the likelihood functions based on image contours and edge features. We remove those particles that do not match the image significantly at the top level, and the remaining particles are evaluated using the underlying likelihood function. The experimental results show that our method can effectively improve the accuracy of motion tracking and constrain the estimation of human body position.

A Robust Monocular 3D Object Tracking Method Combining Statistical and Photometric Constraints

Both region-based methods and direct methods have become popular in recent years for tracking the 6-dof pose of an object from monocular video sequences. Region-based methods estimate the pose of t...

Template-Based 3D Reconstruction of Non-rigid Deformable Object from Monocular Video

In this paper, we propose a template-based 3D surface reconstruction system of non-rigid deformable objects from monocular video sequence. Firstly, we generate a semi-dense template of the target object with structure from motion method using a subsequence video. This video can be captured by rigid moving camera orienting the static target object or by a static camera observing the rigid moving target object. Then, with the reference template mesh as input and based on the framework of classical template-based methods, we solve an energy minimization problem to get the correspondence between the template and every frame to get the time-varying mesh to present the deformation of objects. The energy terms combine photometric cost, temporal and spatial smoothness cost as well as as-rigid-as-possible cost which can enable elastic deformation. In this paper, an easy and controllable solution to generate the semi-dense template for complex objects is presented. Besides, we use an effective iterative Schur based linear solver for the energy minimization problem. The experimental evaluation presents qualitative deformation objects reconstruction results with real sequences. Compare against the results with other templates as input, the reconstructions based on our template have more accurate and detailed results for certain regions. The experimental results show that the linear solver we used performs better efficiency compared to traditional conjugate gradient based solver.

Robust Sparse and Dense Nonrigid Structure From Motion

Modeling deformable three-dimensional (3-D) shape from a video sequence is a fundamental task in computer vision. Nonrigid structure from motion (NRSfM) refers to the problem of recovering the 3-D shape and pose of an object, deforming over time from a monocular video sequence. Presently, dense NRSfM is a research problem of great interest in academia and the industry due to the large demand for 3-D data in various contexts. We provide a robust system for the sparse and dense NRSfM. The strength of our approach comes with the ability to deal with the trajectories corrupted with outliers that serve as an input to NRSfM. To tackle this problem, the input trajectories are processed with a density-based spatial clustering approach that is combined with a RANSAC technique for the outlier' s detection. Processing the trajectories with this process enhances the trajectories by removing the unwanted outliers. Also, extending the work from sparse to dense NRS fM substantially increases the difficulty of the optimization problem. Thus, the proposed system also provides asymptotic improvements to the current optimization approaches by providing an efficient and a novel supervised Gauss-Newton method. Extensive experiments have demonstrated that the proposed method outperforms most of the existing NRSfM methods. The results show that the proposed method reconstructs largely deforming objects accurately and efficiently.

3D facial feature and expression computing from Internet image or video

Large-scale multimedia datasets such as the Internet image and video collections provide new opportunities to understand and analyze human actions, among which one of the most interesting type is facial performance. In this paper, we present an automatic reconstruction system of detailed face performances. Many existing facial performance reconstruction systems rely on data captured under controlled environments with densely spaced cameras and lights. On the contrary, our system reconstructs detailed facial geometry from just one image or a monocular video sequence with unknown lighting. To achieve this, we first simultaneously track 2D and 3D sparse features, then reconstruct the low frequency facial geometry by performing a 2D-3D feature trajectory fusion optimization, which we formulate as a linear problem that can be solved efficiently. Finally, we use a per-pixel shape-from-shading algorithm to estimate the fine-scale geometry details such as wrinkles to further improve the reconstruction fidelity. We demonstrate the accuracy of our system with reconstruction results using both single images and monocular video sequences.

Shot Segmentation for Binocular Stereoscopic Video Based on Spatial–Temporal Feature Clustering

Shot segmentation is the key to content-based analysis, index and retrieval of binocular stereoscopic video. To solve the problem of low accuracy of stereoscopic video shot segmentation in which the segmentation method of 2D video is used to segment monocular video sequence, and the disadvantages of some stereoscopic video shot segmentation methods, a shot segmentation method for binocular stereoscopic video based on spatial---temporal feature clustering (STFC) is proposed. In the method, the features of color and brightness of left video frames in temporal domain as well as the depth feature acquired by matching of left and right frames in spatial domain is extracted. The feature differences between frames are calculated and quantified. Then the clustering of feature differences in three-dimensional space is executed, and the optimization and iteration of the classes are implemented to achieve the division of shot boundary. The experimental results show that the proposed method can effectively solve the problems of error and omission, especially the inaccuracy of smooth shot detection in binocular stereo video shot segmentation when compared with the latest existing algorithm. The higher accuracy of segmentation can be achieved.

Visual Tracking Based on Dynamic Coupled Conditional Random Field Model

This paper proposes a novel approach to visual tracking of moving objects based on the dynamic coupled conditional random field (DcCRF) model. The principal idea is to integrate a variety of relevant knowledge about object tracking into a unified dynamic probabilistic framework, which is called the DcCRF model in this paper. Under this framework, the proposed approach integrates spatiotemporal contextual information of motion and appearance, as well as the compatibility between the foreground label and object label. An approximate inference algorithm, i.e., loopy belief propagation, is adopted to conduct the inference. Meanwhile, the background model is adaptively updated to deal with gradual background changes. Experimental results show that the proposed approach can accurately track moving objects (with or without occlusions) in monocular video sequences and outperforms some state-of-the-art methods in tracking and segmentation accuracy.

Monocular 3D Reconstruction and Augmentation of Elastic Surfaces with Self-Occlusion Handling.

This paper focuses on the 3D shape recovery and augmented reality on elastic objects with self-occlusions handling, using only single view images. Shape recovery from a monocular video sequence is an underconstrained problem and many approaches have been proposed to enforce constraints and resolve the ambiguities. State-of-the art solutions enforce smoothness or geometric constraints, consider specific deformation properties such as inextensibility or resort to shading constraints. However, few of them can handle properly large elastic deformations. We propose in this paper a real-time method that uses a mechanical model and able to handle highly elastic objects. The problem is formulated as an energy minimization problem accounting for a non-linear elastic model constrained by external image points acquired from a monocular camera. This method prevents us from formulating restrictive assumptions and specific constraint terms in the minimization. In addition, we propose to handle self-occluded regions thanks to the ability of mechanical models to provide appropriate predictions of the shape. Our method is compared to existing techniques with experiments conducted on computer-generated and real data that show the effectiveness of recovering and augmenting 3D elastic objects. Additionally, experiments in the context of minimally invasive liver surgery are also provided and results on deformations with the presence of self-occlusions are exposed.