Monocular Image Research Articles

Depth Estimation for Monocular Image Based on Convolutional Neural Networks

Perceiving the three-dimensional structure of the surrounding environment and analyzing it for autonomous movement is an indispensable element for robots to operate in scenes. Recovering depth information and the three-dimensional spatial structure from monocular images is a basic mission of computer vision. For the objects in the image, there are many scenes that may produce it. This paper proposes to use a supervised end-to-end network to perform depth estimation without relying on any subsequent processing operations, such as probabilistic graphic models and other extra fine steps. This paper uses an encoder-decoder structure with feature pyramid to complete the prediction of dense depth maps. The encoder adopts ResNeXt-50 network to achieve main features from the original image. The feature pyramid structure can merge high and low level information with each other, and the feature information is not lost. The decoder utilizes the transposed convolutional and the convolutional layer to connect as an up-sampling structure to expand the resolution of the output. The structure adopted in this paper is applied to the indoor dataset NYU Depth v2 to obtain better prediction results than other methods. The experimental results show that on the NYU Depth v2 dataset, our method achieves the best results on 5 indicators and the sub-optimal results on 1 indicator.

Contrast-reversed binocular dot-pairs in random-dot stereograms for depth perception in central visual field: Probing the dynamics of feedforward-feedback processes in visual inference

In a random-dot stereogram (RDS), the spatial disparities between the interocularly corresponding black and white random dots determine the depths of object surfaces. If a black dot in one monocular image corresponds to a white dot in the other, disparity-tuned neurons in primary visual cortex (V1) respond as if their preferred disparities become non-preferred and vice versa, reversing the disparity sign reported to higher visual areas. Reversed depth is perceptible in the peripheral but not the central visual field. This study demonstrates that, in central vision, adding contrast-reversed dots to a noisy RDS (containing the normal contrast-matched dots) can augment or degrade depth perception. Augmentation occurs when the reversed depth signals are congruent with the normal depth signals to report the same disparity sign, and occurs regardless of the viewing duration. Degradation occurs when the reversed and normal depth signals are incongruent with each other and when the RDS is viewed briefly. These phenomena reflect the Feedforward-Feedback-Verify-and-reWeight (FFVW) process for visual inference in central vision, and are consistent with the central-peripheral dichotomy that central vision has a stronger top-down feedback from higher to lower brain areas to disambiguate noisy and ambiguous inputs from V1. When a RDS is viewed too briefly for feedback, augmentation and degradation work by adding the reversed depth signals from contrast-reversed dots to the feedforward, normal, depth signals. With a sufficiently long viewing duration, the feedback vetoes incongruent reversed depth signals and amends or completes the imperfect, but congruent, reversed depth signals by analysis-by-synthesis computation.

Colonoscopic 3D reconstruction by tubular non-rigid structure-from-motion.

The visual examination of colonoscopic images fails to extract precise geometric information of the colonic surface. Reconstructing the 3D surface of the colon from colonoscopic image sequences may thus add valuable clinical information. We address this problem of extracting precise spatio-temporal 3D structure information from colonoscopic images. Using just the intrinsically calibrated monocular image stream, we develop a technique to compute the depth of certain feature points that have been tracked across images. Our method uses the prior knowledge of an approximate geometry of the colon, called the (TTP). It works by fitting a deformable cylindrical model to points reconstructed independently by non-rigid structure-from-motion (NRSfM), compromising between the data term and a novel tubular smoothing prior. Our method represents the first method ever to exploit a very weak topological prior to improve NRSfM. As such, it lies in-between standard NRSfM, which does not use a topological prior beyond the mere plane, and shape-from-template (SfT), which uses a very strong prior as a full deformable 3D object model. We validate our method on both synthetic images of tubular structures and real colonoscopic data. Our method improves the results obtained by existing NRSfM methods by 71.74% on average on synthetic data and succeeds in obtaining 3D reconstruction from a real colonoscopic sequence defeating the existing methods. Colonoscopic 3D reconstruction is a difficult problem, which is yet unresolved by the existing methods from computer vision. Our proposed dedicated NRSfM method and experiments show that the visual motion might be the right visual cue to use in colonoscopy.

Disparity in Context: Understanding how monocular image content interacts with disparity processing in human visual cortex

Horizontal disparities between the two eyes’ retinal images are the primary cue for depth. Commonly used random ot tereograms (RDS) intentionally camouflage the disparity cue, breaking the correlations between monocular image structure and the depth map that are present in natural images. Because of the nonlinear nature of visual processing, it is unlikely that simple computational rules derived from RDS will be sufficient to explain binocular vision in natural environments. In order to understand the interplay between natural scene structure and disparity encoding, we used a depth-image-based-rendering technique and a library of natural 3D stereo pairs to synthesize two novel stereogram types in which monocular scene content was manipulated independent of scene depth information. The half-images of the novel stereograms comprised either random-dots or scrambled natural scenes, each with the same depth maps as the corresponding natural scene stereograms. Using these stereograms in a simultaneous Event-Related Potential and behavioral discrimination task, we identified multiple disparity-contingent encoding stages between 100 ~ 500 msec. The first disparity sensitive evoked potential was observed at ~100 msec after an earlier evoked potential (between ~50-100 msec) that was sensitive to the structure of the monocular half-images but blind to disparity. Starting at ~150 msec, disparity responses were stereogram-specific and predictive of perceptual depth. Complex features associated with natural scene content are thus at least partially coded prior to disparity information, but these features and possibly others associated with natural scene content interact with disparity information only after an intermediate, 2D scene-independent disparity processing stage.

Deep learning for material synthesis and pose estimation material systems: A review

Abstract The estimation of human pose has been an important field of study for the area of computer vision. Due to its significant applications in various important fields such as human computer interaction, action recognition, video surveillance, threat prediction, etc., it has been in the focus of researchers mainly. Dramatic effects of deep learning techniques have resulted in major steps and important developments in estimating human poses. We are proposing a fast and efficient deep learning approach to detecting 2D pose from a monocular image by multiple persons. The method uses a Vector Field (PVF) part affinity, which learns how well the limbs have been connected. The architectural design provides a global pixel-level interpretation of the number of people by enabling a bottom-up approach that maintains high accuracy in real time. The architecture has been designed to use two branches of a convolutional neural network (CNN) with sequential prediction to understand and reference part locations around each other. Successfully, with the help of the MS COCO human pose dataset, we have implemented our approach.

Practical Face Reconstruction via Differentiable Ray Tracing

AbstractWe present a differentiable ray‐tracing based novel face reconstruction approach where scene attributes – 3D geometry, reflectance (diffuse, specular and roughness), pose, camera parameters, and scene illumination – are estimated from unconstrained monocular images. The proposed method models scene illumination via a novel, parameterized virtual light stage, which in‐conjunction with differentiable ray‐tracing, introduces a coarse‐to‐fine optimization formulation for face reconstruction. Our method can not only handle unconstrained illumination and self‐shadows conditions, but also estimates diffuse and specular albedos. To estimate the face attributes consistently and with practical semantics, a two‐stage optimization strategy systematically uses a subset of parametric attributes, where subsequent attribute estimations factor those previously estimated. For example, self‐shadows estimated during the first stage, later prevent its baking into the personalized diffuse and specular albedos in the second stage. We show the efficacy of our approach in several real‐world scenarios, where face attributes can be estimated even under extreme illumination conditions. Ablation studies, analyses and comparisons against several recent state‐of‐the‐art methods show improved accuracy and versatility of our approach. With consistent face attributes reconstruction, our method leads to several style – illumination, albedo, self‐shadow – edit and transfer applications, as discussed in the paper.

Real time Localization Method Research with Monoslam and BIM Model

Outdoor navigation technologies based on GPS and Beidou has been very mature. However, due to the weakness of satellite signal and refraction, they can’t satisfy the needs of indoor localization and parking area navigation in large commercial buildings. Most indoor localization technologies, such as Bluetooth and Wi-Fi, are based on active equipment. Due to the implementation cost, they can’t be widely deployed. In recent years, SLAM has been greatly developed and applied in the field of robotics and autonomous driving. For technologies such as stereo SLAM and Lidar SLAM require additional equipment, and meanwhile large scale computing will happen during point cloud matching, they are not suitable for civilian mobile usage. Considering the BIM technology, which can provide accurate 3D data of buildings, are widely used gradually, a real-time localization method combining monocular SLAM and BIM model is proposed to complete vision-based initial localization. First, we create the BIM model with the design drawings, extract the visible edge lines of model, and establish a 3D line feature database; then, intercept the key frames of the mobile camera, and extract the edge line and line feature descriptors from image, match the line features between key frames, triangulate the matched line segments, get the 3D line coordinate in first frame, mistake filter will be rolled out between multiple frames; finally, match the reconstruction result with line feature database form BIM model, register the initial position. This method can quickly and effectively perform the registration of monocular images and BIM models, get the initial position in complex spaces, and support the application scenarios such as indoor navigation, equipment maintenance, facility management, and construction monitoring.

DIMNet: Dense implicit function network for 3D human body reconstruction

In recent years, with the improvement of artificial intelligence technology, it has become possible to reconstruct high-precision 3D human body models based on ordinary RGB images. The current 3D human body reconstruction technology requires complex external equipment to scan all angles of the human body, which is complicated to be implemented and cannot be popularized. In order to solve this problem, this paper applies deep learning models on reconstructing 3D human body based on monocular images. First of all, this paper uses Stacked Hourglass network to perform convolution operations on monocular images collected from different views. Then Multi-Layer Perceptrons (MLPs) are used to decode the encoded high-level images. The feature codes in the two views(main and side) are fused, and the interior and exterior points are classified by the fusion features, so as to obtain the corresponding 3D occupancy field. At last, the Marching Cube algorithm is used for 3D reconstruction with a specific threshold and then we use Laplace smoothing algorithm to remove artifacts. This paper proposes a dense sampling strategy based on the important joint points of the human body, which has a certain optimization effect on the realization of high-precision 3D reconstruction. The performance of the proposed scheme has been validated on the open source datasets, MGN dataset and the THuman dataset, provided by Tsinghua University. The proposed scheme can reconstruct features such as clothing folds, color textures, and facial details,and has great potential to be applied in different applications.

Sculpture 3D Modeling Method Based on Image Sequence

This thesis first introduces the basic principles of model-based image sequence coding technology, then discusses in detail the specific steps in various implementation algorithms, and proposes a basic feature point calibration required in three-dimensional motion and structure estimation. This is a simple and effective solution. Aiming at the monocular video image sequence obtained by only one camera, this paper introduces the 3D model of the sculpture building into the pose tracking framework to provide initial depth information. The whole posture tracking framework can be divided into three parts, namely, the construction of the initial sculpture model, the posture tracking between frames, and the robustness processing during continuous tracking. In order to reduce the complexity of calculation, this paper proposes a new three-dimensional mesh model and a moving image restoration algorithm based on this model. At the same time, the influence of the intensity and direction factors in the scene is added, the simulation results are given, and the next step is discussed. The optimization work that needs to be done.

Instance weighting through data imprecisiation

In machine learning, instance weighting is commonly used to control the influence of individual data points in a learning process. The general idea is to improve results (e.g., the accuracy of a predictor) by restricting the influence of training examples that do not appear to be representative and may bias the learner in an undesirable way. The simplest and most common approach is to modulate the influence of each data point through multiplicative scaling. In this paper, we elaborate on the idea of instance weighting through data imprecisiation as a viable alternative to existing methods, and formalize this approach within the framework of superset learning. Roughly speaking, the idea is to reduce the influence of training examples by turning a precise data point into an imprecise observation. Within the framework of optimistic superset learning, a generic approach to superset learning, this effectively comes down to modifying an underlying loss function on a per-instance basis. We illustrate our approach for the case of binary classification with support vector machines, showing that it compares favorably with existing approaches to instance weighting in support vector machines. In a further case study, we demonstrate the usefulness of instance weighting through data imprecisiation for the practical problem of depth estimation in monocular images.

Recovering dense 3D point clouds from single endoscopic image

Background and objective: Recovering high-quality 3D point clouds from monocular endoscopic images is a challenging task. This paper proposes a novel deep learning-based computational framework for 3D point cloud reconstruction from single monocular endoscopic images.Methods: An unsupervised mono-depth learning network is used to generate depth information from monocular images. Given a single mono endoscopic image, the network is capable of depicting a depth map. The depth map is then used to recover a dense 3D point cloud. A generative Endo-AE network based on an auto-encoder is trained to repair defects of the dense point cloud by generating the best representation from the incomplete data. The performance of the proposed framework is evaluated against state-of-the-art learning-based methods. The results are also compared with non-learning based stereo 3D reconstruction algorithms.Results: Our proposed methods outperform both the state-of-the-art learning-based and non-learning based methods for 3D point cloud reconstruction. The Endo-AE model for point cloud completion can generate high-quality, dense 3D endoscopic point clouds from incomplete point clouds with holes. Our framework is able to recover complete 3D point clouds with the missing rate of information up to 60%. Five large medical in-vivo databases of 3D point clouds of real endoscopic scenes have been generated and two synthetic 3D medical datasets are created. We have made these datasets publicly available for researchers free of charge.Conclusions: The proposed computational framework can produce high-quality and dense 3D point clouds from single mono-endoscopy images for augmented reality, virtual reality and other computer-mediated medical applications.

MOLTR: Multiple Object Localization, Tracking and Reconstruction From Monocular RGB Videos

Semantic aware reconstruction is more advantageous than geometric-only reconstruction for future robotic and AR/VR applications because it represents not only where things are, but also what things are. Object-centric mapping is a task to build an object-level reconstruction where objects are separate and meaningful entities that convey both geometry and semantic information. In this letter, we present MOLTR, a solution to object-centric mapping using only monocular image sequences and camera poses. It is able to localize, track and reconstruct multiple rigid objects in an online fashion when a RGB camera captures a video of the surrounding. Given a new RGB frame, MOLTR firstly applies a monocular 3D detector to localize objects of interest and extract their shape codes that represent the object shape in a learnt embedding space. Detections are then merged to existing objects in the map after data association. Motion state (i.e., kinematics and the motion status) of each object is tracked by a multiple model Bayesian filter and object shape is progressively refined by fusing multiple shape code. We evaluate localization, tracking and reconstruction on benchmarking datasets for indoor and outdoor scenes, and show superior performance over previous approaches.

Collision Avoidance by Identifying Risks for Detected Objects in Autonomous Vehicles

We propose a system which will detect objects onour roads, estimate the distance of these object from the cameraand alert the driver if this distance is equal or less than thethreshold value(02meters),and assist the driver and alert him assoon as possible in order for him to take appropriate actions assoon as possible which can avoid any collision or significantlyreduce it. We plan to use state of the arts object detection modelslike YOLO to identify the target object classes and use depthmaps from monocular camera to be give an accurate estimate ofthe distance of the detected object from the camera. one majorrequirement of this system is the real-time behaviour and a highaccuracy for the detected and estimated distance, A secondrequirement is to make the system cheap and easy useablecomparatively to the other existing methods. That is why wedecided to use monocular camera images and depth maps whichmakes the solution cheap and innovative. This project(prototype) provide room for bigger and more complete projectwhich will contribute to the creation of tool which can save livesand improve security on our roads

Measuring Canopy Geometric Structure Using Optical Sensors Mounted on Terrestrial Vehicles: A Case Study in Vineyards

Smart and precision agriculture concepts require that the farmer measures all relevant variables in a continuous way and processes this information in order to build better prescription maps and to predict crop yield. These maps feed machinery with variable rate technology to apply the correct amount of products in the right time and place, to improve farm profitability. One of the most relevant information to estimate the farm yield is the Leaf Area Index. Traditionally, this index can be obtained from manual measurements or from aerial imagery: the former is time consuming and the latter requires the use of drones or aerial services. This work presents an optical sensing-based hardware module that can be attached to existing autonomous or guided terrestrial vehicles. During the normal operation, the module collects periodic geo-referenced monocular images and laser data. With that data a suggested processing pipeline, based on open-source software and composed by Structure from Motion, Multi-View Stereo and point cloud registration stages, can extract Leaf Area Index and other crop-related features. Additionally, in this work, a benchmark of software tools is made. The hardware module and pipeline were validated considering real data acquired in two vineyards—Portugal and Italy. A dataset with sensory data collected by the module was made publicly available. Results demonstrated that: the system provides reliable and precise data on the surrounding environment and the pipeline is capable of computing volume and occupancy area from the acquired data.

Fully Automatic Multi-Object Articulated Motion Tracking

Fully automatic tracking of articulated motion in real-time with a monocular RGB camera is a challenging problem which is essential for many virtual reality (VR) and human-computer interaction applications. In this paper, we present an algorithm for multiple articulated objects tracking based on monocular RGB image sequence. Our algorithm can be directly employed in practical applications as it is fully automatic, real-time, and temporally stable. It consists of the following stages: dynamic objects counting, objects specific 3D skeletons generation, initial 3D poses estimation, and 3D skeleton fitting which fits each 3D skeleton to the corresponding 2D body-parts locations. In the skeleton fitting stage, the 3D pose of every object is estimated by maximizing an objective function that combines a skeleton fitting term with motion and pose priors. To illustrate the importance of our algorithm for practical applications, we present competitive results for real-time tracking of multiple humans. Our algorithm detects objects that enter or leave the scene, and dynamically generates or deletes their 3D skeletons. This makes our monocular RGB method optimal for real-time applications. We show that our algorithm is applicable for tracking multiple objects in outdoor scenes, community videos, and low-quality videos captured with mobile-phone cameras. Keywords: Multi-object motion tracking, Articulated motion capture, Deep learning, Anthropometric data, 3D pose estimation. DOI: 10.7176/CEIS/12-1-01 Publication date: March 31 st 2021

3D Object Detection Based on Proposal Generation Network Utilizing Monocular Images

Monocular 3-D object detection is a low-cost and challenging task for autonomous vehicles and robotics. Utilizing a monocular image for 3-D object detection is served as an auxiliary module for autonomous vehicles and is a growing concern recently. Currently, the expensive lidar and stereo cameras have a predominant performance on accurate 3-D object detection, whereas monocular-based methods are considerably lower in performance. This performance gap is minimized by reforming the monocular-based method as a single internal network. We exploit the correlation between 2-D and 3-D detection spaces, enabling 3-D boxes to leverage feature maps generated in image space. The 2-D and 3-D proposals are extracted through a proposal generation network that is enhanced and utilized for estimating accurate 3-D detection and localization. Experimental results on the KITTI dataset demonstrate that in comparison to other monocular object detection methods the proposed method considerably improved the accuracy of 3-D object detection. The mean average precision of 3-D object detection in front view is improved to 25% and the bird's eye view to 32% for the car class on a moderate difficulty level.

Perceptual Monocular Depth Estimation

Monocular depth estimation (MDE), which is the task of using a single image to predict scene depths, has gained considerable interest, in large part owing to the popularity of applying deep learning methods to solve “computer vision problems”. Monocular cues provide sufficient data for humans to instantaneously extract an understanding of scene geometries and relative depths, which is evidence of both the processing power of the human visual system and the predictive power of the monocular data. However, developing computational models to predict depth from monocular images remains challenging. Hand-designed MDE features do not perform particularly well, and even current “deep” models are still evolving. Here we propose a novel approach that uses perceptually-relevant natural scene statistics (NSS) features to predict depths from monocular images in a simple, scale-agnostic way that is competitive with state-of-the-art systems. While the statistics of natural photographic images have been successfully used in a variety of image and video processing, analysis, and quality assessment tasks, they have never been applied in a predictive end-to-end deep-learning model for monocular depth. Correspondingly, no previous work has explicitly incorporated perceptual features in a monocular depth-prediction approach. Here we accomplish this by developing a new closed-form bivariate model of image luminances and use features extracted from this model and from other NSS models to drive a novel deep learning framework for predicting depth given a single image.

Building a Real-Time 2D Lidar Using Deep Learning

Applying deep learning methods, this paper addresses depth prediction problem resulting from single monocular images. A vector of distances is predicted instead of a whole image matrix. A vector-only prediction decreases training overhead and prediction periods and requires less resources (memory, CPU). We propose a module which is more time efficient than the state-of-the-art modules ResNet, VGG, FCRN, and DORN. We enhanced the network results by training it on depth vectors from other levels (we get a new level by changing the Lidar tilt angle). The predicted results give a vector of distances around the robot, which is sufficient for the obstacle avoidance problem and many other applications.

Depth estimation for a robotic arm based on monocular images

Regaining depth information of objects from two-dimensional images is one of the fundamental issues and essential thing in the field of machine vision. There is a depth estimation method using non-inclined with forward movements that was already presented with feature extraction of image processing based on one direction only. This study proposes a depth estimation approach based on forward and backward movement with non-inclined and inclined orientations. Binary threshold will be used in this study to replace feature extraction. The study can approximate the most effective depth estimation using forward and backward movement in frame of non-inclined and inclined orientations. The images captured by an un-calibrated ordinary monocular camera mounted at the end effector of a robot arm. In this approach, the first image is captured and then the camera parameters remain unchanged. The second image is acquired after moving a distance d along the optical axis. Then image segmentation and binary threshold are implemented on the two images separately, and the numbers of black pixels in the images are counted. The experiments were carried out by simulations using webot software and python programming code. The results showed that the second position of the camera 0.1 meter from its first position demonstrated the best estimation performance, both for non-inclined and inclined movements, while the 0.01 meter distance resulted in the greatest error.

Self-Supervised Face Image Manipulation by Conditioning GAN on Face Decomposition

We present a novel architecture for manipulating facial expressions, head poses, and lighting conditions from a single monocular image. Recent methods based on Generative Adversarial Networks show promising results in expression manipulation. However, the variation is either defined by a limited number of classes or not well suitable for explicit manipulation of different attributes such as pose and lighting conditions. Besides, state-of-the-art methods are mostly focused on frontal faces. Therefore, in this paper, a new Generative Adversarial Network architecture is proposed by explicitly conditioning on the appearance image space which is the product of direct manipulation of facial expressions, light and pose conditions of the face model in 3D space. In addition, the method only requires video sequences for training. Therefore, it is self-supervised. Unlike other face manipulation methods, the proposed method does not require target specific training. Large scale experiments show that our method outperforms state-of-the-art methods for different scenarios.