Monocular Image Sequences Research Articles

Prior-free 3D human pose estimation in a video using limb-vectors

Estimating accurate 3D human poses from a monocular video is fundamental to various computer vision tasks. Existing methods exploit 2D-to-3D pose lifting, multiview images, and depth sensors to model spatio-temporal dependencies. However, depth ambiguities, occlusions, and larger temporal receptive fields pose challenges to these approaches. To address this, we propose a novel prior-free DCNN-based 3D human pose estimation method for monocular image sequences using limb vectors. Our method comprises two subnetworks: a limb direction estimator and a limb length estimator. The limb direction estimator utilizes a fully convolutional network to model limb direction vectors across a temporal window. We show that network complexity can be significantly reduced by utilizing dilated convolutional operations and a relatively smaller receptive field while maintaining estimation accuracy. Moreover, the limb length estimator captures stable limb length estimations from a reliable frame set. Our model has shown superior performance compared to existing methods on the Human3.6M and MPI-INF-3DHP datasets.

Image‐based analysis of fresh concrete flow ‐ determining the correlation between flow behavior and rheological properties

AbstractBesides compressive strength, the workability of fresh concrete is one of the most important characteristics of concrete. Only by adapting the rheology of the fresh concrete to the geometry of the element to be casted, the formation of air voids and damages can be avoided. This holds even more so true for 3D printable concrete. Currently, empirical test methods such as the slump test are used to determine rheological parameters. Rheometer tests, on the other hand, allow a much deeper insight into the rheological properties of concrete but are more challenging. Further, all currently available test methods can only be applied on a batch basis.In the paper at hand, a new approach for an automatic digital concrete quality control is presented using modern computer‐vision based analysis methods. Using a newly developed experimental setup, the flow of mortar down an inclined open channel is studied using a monocular camera setup. Differences in the mortar's rheology result in different flow behaviors which can be optically detected. Utilizing dense optical flow, rheological parameters are determined from the acquired monocular image sequences. The results shown in this paper prove that the computed flow behavior of the investigated mortars precisely correlates with their rheological properties, demonstrating the high potential of the method for an automated in‐line testing of fresh concrete e.g. during the discharge of a mixing truck.

Automatic endoscopic navigation based on attention-based network for Nasotracheal Intubation

Nasotracheal intubation (NTI) is one of the most commonly performed procedures in anesthesia and is considered the gold standard for securing the airway of patients. Endoscope operation is critical to the success of NTI. However, this operation remains challenging as it requires the surgeon to classify anatomical landmarks and detect heading targets of the endoscope tip in a sequence of monocular images. To alleviate this problem, this study presents a learning-based navigation method that automatically classifies four different anatomical landmarks and detects the heading target of the endoscope tip from endoscopic images. First, an end-to-end multitask network is introduced that consists of a branch for anatomical landmark classification and another for heading target detection. In addition, a convolutional attention module is designed to improve network performance by combining spatial and channel attention. Second, an endoscopic dataset named intuNav is built for network training. The trained network calculates navigation information without image prior knowledge. Finally, extensive experiments on the built dataset and endoscopic videos demonstrate the high performance of our method, achieving classification (94%) and detection (79.4%) accuracies. The results also indicate that the proposed method is effective and efficient in navigation generation during NTI.

Motion Estimation of Non-Cooperative Space Objects Based on Monocular Sequence Images

The motion estimation of non-cooperative space objects is an important prerequisite for successfully completing on-orbit servicing, debris removing and asteroid exploration missions. The motion estimation of non-cooperative space objects is a challenging task because of the lack of prior information about the object in unknown scenarios. In this paper, a motion estimation method of non-cooperative space objects for autonomous navigation based on a monocular sequence of images is proposed. The rotational attitude estimation of the non-cooperative space object is realized by matching the feature points of adjacent frame images, and the conditions required for translational estimation are discussed. This method has small calculation costs and runs stably, which meets the requirements of real-time computation on-board. As a result, this method implements the estimation of the complete unknown information of the object, including the attitude quaternion and angular velocity. Numerical simulations and results verify the effectiveness of the proposed method.

Imposing temporal consistency on deep monocular body shape and pose estimation

Accurate and temporally consistent modeling of human bodies is essential for a wide range of applications, including character animation, understanding human social behavior, and AR/VR interfaces. Capturing human motion accurately from a monocular image sequence remains challenging; modeling quality is strongly influenced by temporal consistency of the captured body motion. Our work presents an elegant solution to integrating temporal constraints during fitting. This increases both temporal consistency and robustness during optimization. In detail, we derive parameters of a sequence of body models, representing shape and motion of a person. We optimize these parameters over the complete image sequence, fitting a single consistent body shape while imposing temporal consistency on the body motion, assuming body joint trajectories to be linear over short time. Our approach enables the derivation of realistic 3D body models from image sequences, including jaw pose, facial expression, and articulated hands. Our experiments show that our approach accurately estimates body shape and motion, even for challenging movements and poses. Further, we apply it to the particular application of sign language analysis, where accurate and temporally consistent motion modelling is essential, and show that the approach is well-suited to this kind of application.

Sculpture 3D Modeling Method Based on Image Sequence

This thesis first introduces the basic principles of model-based image sequence coding technology, then discusses in detail the specific steps in various implementation algorithms, and proposes a basic feature point calibration required in three-dimensional motion and structure estimation. This is a simple and effective solution. Aiming at the monocular video image sequence obtained by only one camera, this paper introduces the 3D model of the sculpture building into the pose tracking framework to provide initial depth information. The whole posture tracking framework can be divided into three parts, namely, the construction of the initial sculpture model, the posture tracking between frames, and the robustness processing during continuous tracking. In order to reduce the complexity of calculation, this paper proposes a new three-dimensional mesh model and a moving image restoration algorithm based on this model. At the same time, the influence of the intensity and direction factors in the scene is added, the simulation results are given, and the next step is discussed. The optimization work that needs to be done.

MOLTR: Multiple Object Localization, Tracking and Reconstruction From Monocular RGB Videos

Semantic aware reconstruction is more advantageous than geometric-only reconstruction for future robotic and AR/VR applications because it represents not only where things are, but also what things are. Object-centric mapping is a task to build an object-level reconstruction where objects are separate and meaningful entities that convey both geometry and semantic information. In this letter, we present MOLTR, a solution to object-centric mapping using only monocular image sequences and camera poses. It is able to localize, track and reconstruct multiple rigid objects in an online fashion when a RGB camera captures a video of the surrounding. Given a new RGB frame, MOLTR firstly applies a monocular 3D detector to localize objects of interest and extract their shape codes that represent the object shape in a learnt embedding space. Detections are then merged to existing objects in the map after data association. Motion state (i.e., kinematics and the motion status) of each object is tracked by a multiple model Bayesian filter and object shape is progressively refined by fusing multiple shape code. We evaluate localization, tracking and reconstruction on benchmarking datasets for indoor and outdoor scenes, and show superior performance over previous approaches.

Fully Automatic Multi-Object Articulated Motion Tracking

Fully automatic tracking of articulated motion in real-time with a monocular RGB camera is a challenging problem which is essential for many virtual reality (VR) and human-computer interaction applications. In this paper, we present an algorithm for multiple articulated objects tracking based on monocular RGB image sequence. Our algorithm can be directly employed in practical applications as it is fully automatic, real-time, and temporally stable. It consists of the following stages: dynamic objects counting, objects specific 3D skeletons generation, initial 3D poses estimation, and 3D skeleton fitting which fits each 3D skeleton to the corresponding 2D body-parts locations. In the skeleton fitting stage, the 3D pose of every object is estimated by maximizing an objective function that combines a skeleton fitting term with motion and pose priors. To illustrate the importance of our algorithm for practical applications, we present competitive results for real-time tracking of multiple humans. Our algorithm detects objects that enter or leave the scene, and dynamically generates or deletes their 3D skeletons. This makes our monocular RGB method optimal for real-time applications. We show that our algorithm is applicable for tracking multiple objects in outdoor scenes, community videos, and low-quality videos captured with mobile-phone cameras. Keywords: Multi-object motion tracking, Articulated motion capture, Deep learning, Anthropometric data, 3D pose estimation. DOI: 10.7176/CEIS/12-1-01 Publication date: March 31 st 2021

Self-Supervised Monocular Depth Estimation With Extensive Pretraining

Although depth estimation is a key technology for three-dimensional sensing applications involving motion, active sensors such as LiDAR and depth cameras tend to be expensive and bulky. Here, we explore the potential of monocular depth estimation (MDE) using a self-supervised approach. MDE is a promising technology, but supervised learning suffers from a need for accurate ground-truth depth data. Recent studies have enabled self-supervised training on an MDE model with only monocular image sequences and image-reconstruction errors. We pretrained networks using multiple datasets, including monocular and stereo image sequences. The main challenges posed by the self-supervised MDE model were occlusions and dynamic objects. We proposed novel loss functions to handle these problems in the form of min-over-all and min-with-flow losses, both based on the per-pixel minimum reprojection error of Monodepth2 and extended to stereo images and optical flow. With extensive pretraining and novel losses, our model outperformed existing unsupervised approaches in quantitative depth estimation and the ability to distinguish small objects against a background, as evaluated by KITTI 2015.

Visual Landmark Learning Via Attention-Based Deep Neural Networks

Compared with such high-precision and dense representation maps as the point cloud or occupancy grid maps, the landmark map has the advantages of compactness and memory efficiency, where these advantages are particularly prominent in a large-scale environment for robot localization, navigation, or environment measurement. However, training a robot to identify and select useful landmarks for localization is challenging. Due to this limitation, most landmarks are identified using handcrafted features. In this article, we propose a multitask neural network called LandmarkNet to build a flexible, optimal, and interpretable landmark map. The multitask neural network with an attention mechanism is trained for the robot’s pose regression and the semantic segmentation of images from a set of sequences of monocular images. The image segmentation yields auxiliary semantic information for landmark selection based on some common-sense notions such as that the sky and clouds cannot be used as landmarks for robot localization. The attention feature map thus learned is used to select landmarks for robot localization based on the optimal projection between the image sequence with the corresponding pose of the robot and the semantic segmentation of the images. The learned landmark maps are also applied to mobile robot localization using traditional Monte Carlo localization (MCL). To verify the validity of our methods, we performed experiments on simulation, an open dataset (KITTI monoVO), and a dataset that we had created on the campus of Tongji University. The results show that the learned landmark map, which was only 6% of the size of the original map, can be used to accurately perform visual localization tasks. We open the source code on website https://github.com/dotstudio01/landmark_learning.git .

DeepSLAM: A Robust Monocular SLAM System With Unsupervised Deep Learning

In this article, we propose DeepSLAM, a novel unsupervised deep learning based visual simultaneous localization and mapping (SLAM) system. The DeepSLAM training is fully unsupervised since it only requires stereo imagery instead of annotating ground-truth poses. Its testing takes a monocular image sequence as the input. Therefore, it is a monocular SLAM paradigm. DeepSLAM consists of several essential components, including Mapping-Net, Tracking-Net, Loop-Net, and a graph optimization unit. Specifically, the Mapping-Net is an encoder and decoder architecture for describing the 3-D structure of environment, whereas the Tracking-Net is a recurrent convolutional neural network architecture for capturing the camera motion. The Loop-Net is a pretrained binary classifier for detecting loop closures. DeepSLAM can simultaneously generate pose estimate, depth map, and outlier rejection mask. In this article, we evaluate its performance on various datasets, and find that DeepSLAM achieves good performance in terms of pose estimation accuracy, and is robust in some challenging scenes.

Monocular 3D Facial Expression Features for Continuous Affect Recognition

Automated facial expression analysis from image sequences for continuous emotion recognition is a very challenging task due to the loss of the three-dimensional information during the image formation process. State-of-the-art relied on estimating dynamic textures features and convolutional neural network features to derive spatio-temporal features. Despite their great success, such features are insensitive to micro facial muscle deformations and are affected by identity, face pose, illumination variation, and self-occlusion. In this work, we argue that retrieving, from image sequences, 3D facial spatio-temporal information, which describes the natural facial muscle deformation, provides a semantical and efficient way of representation and is useful for emotion recognition. In this paper, we propose a framework for extracting three-dimensional facial spatio-temporal features from monocular image sequences using an extended 3D Morphable Model (3DMM) which disentangles the identity factor from the facial expressions of a specific person. An LSTM model is used to evaluate the effectiveness of the proposed spatio-temporal features on video-based facial expression recognition task and continuous affect recognition task. Experimental results, on the AFEW6.0 datasets for facial expression recognition, and the RECOLA and SEMAINE datasets for continuous emotion prediction, illustrate the potential of the proposed 3D spatio-temporal features for facial expressions analysis and continuous affect recognition, as well as their efficiency compared to recent state-of-the-art features.

Deep-learning based reconstruction of the stomach from monoscopic video data

Abstract For the gastroscopic examination of the stomach, the restricted field of view related to the „keyhole“-perspective of the endoscope is known to be a visual limitation. Thus, a panoramic extension can enlarge the field of vision, supports the endoscopist during the examination, and ensures that all of the inner stomach walls are visually inspected. To compute such a panorama of the stomach, knowledge about the geometry of the underlying structure is required. Structure from motion an approach to reconstruct the necessary information about the 3D-structure from monocular image sequences as provided by a gastroscope. We examine and evaluate an existing deep neuronal network for stereo reconstruction, in order to approximate the geometry of stomach parts from a set of consecutive acquired image pairs from gastroscopic videos.

Comparative Study of 3D Reconstruction Methods from 2D Sequential Images in Sports

The process of 3D reconstruction is a basic problem in Computer Vision. However, recent researches have been successfully addressed by motion capture systems with body worn markers and multiple cameras. To recover 3Dreconstruction from fully-body human pose by single camera still remains a challenging problem. For instance, noisy background, variation in human appearance and self-occlusion were among these challenges. This thesis investigated methods of 3D reconstruction from monocular image sequences in dynamic activities such as sports. Six recent methods were selected based on they focused on recovery fully automated system for estimating 3D human pose for 2D joint location. These researches have been developed the algorithm that be able to solve ill-posed problem. Evaluation of the methods was divided in two sections. First, the theoretical and comparative study of each method was disclosed to identify the technique used, the problems that enquired and the results achieved in their approach. After that, the advantages and disadvantages of each method were listed. Also, several factors such as accuracy, self-occlusion and so on have been compared amongst these methods. In Second stage, based on the advantages found in the first stage of evaluation, three methods were chosen to be evaluated using specific data set. Initially, the codes of the three methods on PennAction dataset (tennis) were run and the performance of the methods in 3D reconstruction is showed. Then, the methods were tested on a mixed activities sequence from the CMU motion capture database. The novel of this study is evaluation of recent methods based on the accuracy of their performance on the specific dataset of tennis player. Also, we proposed a technique which combining specific advantages of each method to create a more efficient method for 3D reconstruction of 2D sequential images in the context of outdoor activities.

Depth estimation for a road scene using a monocular image sequence based on fully convolutional neural network

An advanced driving assistant system is one of the most popular topics nowadays, and depth estimation is an important cue for advanced driving assistant system. Depth prediction is a key problem in understanding the geometry of a road scene for advanced driving assistant system. In comparison to other depth estimation methods using stereo depth perception, determining depth relation using a monocular camera is considerably challenging. In this article, a fully convolutional neural network with skip connection based on a monocular video sequence is proposed. With the integration framework that combines skip connection, fully convolutional network and the consistency between consecutive frames of the input sequence, high-resolution depth maps are obtained with lightweight network training and fewer computations. The proposed method models depth estimation as a regression problem and trains the proposed network using a scale invariance optimization based on L2 loss function, which measures the relationships between points in the consecutive frames. The proposed method can be used for depth estimation of a road scene without the need for any extra information or geometric priors. Experiments on road scene data sets demonstrate that the proposed approach outperforms previous methods for monocular depth estimation in dynamic scenes. Compared with the currently proposed method, our method has achieved good results when using the Eigen split evaluation method. The obvious prominent one is that the linear root mean squared error result is 3.462 and the δ < 1.25 result is 0.892.

Deep Direct Visual Odometry

Traditional monocular direct visual odometry (DVO) is one of the most famous methods to estimate the ego-motion of robots and map environments from images simultaneously. However, DVO heavily relies on high-quality images and accurate initial pose estimation during tracking. With the outstanding performance of deep learning, previous works have shown that deep neural networks can effectively learn 6-DoF (Degree of Freedom) poses between frames from monocular image sequences in the unsupervised manner. However, these unsupervised deep learning-based frameworks cannot accurately generate the full trajectory of a long monocular video because of the scale-inconsistency between each pose. To address this problem, we use several geometric constraints to improve the scale-consistency of the pose network, including improving the previous loss function and proposing a novel scale-to-trajectory constraint for unsupervised training. We call the pose network trained by the proposed novel constraint as TrajNet. In addition, a new DVO architecture, called deep direct sparse odometry (DDSO), is proposed to overcome the drawbacks of the previous direct sparse odometry (DSO) framework by embedding TrajNet. Extensive experiments on the KITTI dataset show that the proposed constraints can effectively improve the scale-consistency of TrajNet when compared with previous unsupervised monocular methods, and integration with TrajNet makes the initialization and tracking of DSO more robust and accurate.

Cycle-SfM: Joint self-supervised learning of depth and camera motion from monocular image sequences.

Understanding 3D scene geometry is a fundamental research topic in computer vision, including various subproblems, such as depth prediction, visual odometry, optical flow, etc. With the advent of artificial intelligence methods like deep learning, many approaches have emerged to deal with such problems in an end-to-end manner. These pipelines take the 3D understanding task as a nonlinear optimization problem, with the purpose of minimizing the cost function of the whole framework. Here, we present a self-supervised framework for jointly learning the monocular depth and camera's ego-motion from unlabeled, unstructured, and monocular video sequences. We propose a forward-backward consistency constraint on view reconstruction to capture temporal relations across adjacent frames, whose purpose is to explore and make full use of the bidirectional projection information. A simple and practicable improvement on the design of cost function is proposed to enhance the estimated accuracy. Due to the fact that our improvement is a lightweight and general module, it can be integrated into any self-supervised architectures seamlessly, and more accurate results can be obtained. The evaluation on the KITTI dataset demonstrates that our approach is highly efficient and performs better than the existing works in pose estimation, while the results in depth estimation perform comparably with the existing ones.

Depth Estimation with Ego-Motion Assisted Monocular Camera

We propose a method to estimate the distance to objects based on the complementary nature of monocular image sequences and camera kinematic parameters. The fusion of camera measurements with the kinematics parameters that are measured by an IMU and an odometer is performed using an extended Kalman filter. Results of field experiments with a wheeled robot corroborated the results of the simulation study in terms of accuracy of depth estimation. The performance of the approach in depth estimation is strongly affected by the mutual observer and feature point geometry, measurement accuracy of the observer’s motion parameters and distance covered by the observer. It was found that under favorable conditions the error in distance estimation can be as small as 1% of the distance to a feature point. This approach can be used to estimate distance to objects located hundreds of meters away from the camera.

Research on Pose Estimation of Mobile Robot Based on Convolutional Neural Network

In this paper, we propose a new learning scheme for generating camera pose to be used for visual odometry. Pose estimation of mobile robot in unknown environment is formulated as a learning problem. We train a convolutional neural network end to end to compute the 6-dof pose of the robot from a series of images. The architecture is a kind of residual network with multiple attention resblocks, which can give features different weights in order to improve the accuracy of pose prediction. The network estimates ego-motion taking monocular image sequences as input instead of separate images. We call this novel network architecture Posenet. Compared to traditional methods, results are more accurate and in which we can see the potential of pose estimation methods with convolutional neural network.

Comparative Study of Relative-Pose Estimations from a Monocular Image Sequence in Computer Vision and Photogrammetry.

Techniques for measuring the position and orientation of an object from corresponding images are based on the principles of epipolar geometry in the computer vision and photogrammetric fields. Contributing to their importance, many different approaches have been developed in computer vision, increasing the automation of the pure photogrammetric processes. The aim of this paper is to evaluate the main differences between photogrammetric and computer vision approaches for the pose estimation of an object from image sequences, and how these have to be considered in the choice of processing technique when using a single camera. The use of a single camera in consumer electronics has enormously increased, even though most 3D user interfaces require additional devices to sense 3D motion for their input. In this regard, using a monocular camera to determine 3D motion is unique. However, we argue that relative pose estimations from monocular image sequences have not been studied thoroughly by comparing both photogrammetry and computer vision methods. To estimate motion parameters characterized by 3D rotation and 3D translations, estimation methods developed in the computer vision and photogrammetric fields are implemented. This paper describes a mathematical motion model for the proposed approaches, by differentiating their geometric properties and estimations of the motion parameters. A precision analysis is conducted to investigate the main characteristics of the methods in both fields. The results of the comparison indicate the differences between the estimations in both fields, in terms of accuracy and the test dataset. We show that homography-based approaches are more accurate than essential-matrix or relative orientation–based approaches under noisy conditions.