Monocular Image Research Articles

A transferable approach for quantifying benthic fish sizes and densities in annotated underwater images

Abstract Benthic fishes are a common target of scientific monitoring but are difficult to quantify because of their close association to bottom habitats that are hard to access. Advances in image‐acquisition technologies, machine vision and deep learning have made capturing and quantifying fishes with cameras increasingly feasible. We present a method and open‐source software called ‘FishScale’ to estimate benthic fish lengths, numeric abundance and biomass density in underwater environments assessed with down‐looking monocular images. ‘FishScale’ estimates fish abundances and size frequencies from near‐nadir monocular images where fish have already been semantically segmented. The software accounts for lens distortion, underwater magnification effects and fish body curvature to automatically estimate fish lengths and the areas of images where they were captured. Numeric and biomass density are estimated through a deterministic machine vision algorithm that requires a user‐provided length‐weight relationship for species of interest and calibration images. Results from validation studies show that lengths and weights can be estimated with high accuracy and precision for round goby (Neogobius melanostomus) captured in distorted action camera images, and from large‐bodied lake trout (Salvelinus namaycush) imaged with a machine vision camera. The real‐world utility of the approach is demonstrated in a case study estimating round goby abundances and size frequencies along a 10.7‐km transect surveyed with an autonomous underwater vehicle in Lake Michigan, USA. Our validation studies demonstrate that the approach estimates benthic and benthopelagic fish lengths and weights with little bias and good accuracy and precision for species with much different body shapes and sizes. The method is applicable to data collected using a variety of nadir imaging approaches with widespread applications to fisheries monitoring and quantification of any species or object for which nadir images and working distances between the camera and feature of interest are available.

A monocular three-dimensional object detection model based on uncertainty-guided depth combination for autonomous driving

Three-Dimensional (3D) object detection is a crucial task for enhancing safety and efficiency in autonomous driving. However, estimating depth from monocular images remains a challenging task. Most existing monocular 3D object detection methods rely on additional auxiliary data sources to compensate for the lack of spatial information in monocular images. Nevertheless, these methods bring substantial computational overhead and time-consuming preprocessing steps. To address this issue, we propose a novel depth estimation method for monocular images that does not rely on any auxiliary information. Leveraging both the texture and geometric cues of detected objects, our method generates two depth estimates for each object based on the extracted Region of Interest (RoI) features: a direct depth estimate and a height-based depth estimate with uncertainty modeling. Our model dynamically assigns weights to these depth estimates based on their respective uncertainties and combines them to obtain the final depth. During the training process, the model assigns higher weights to depth branches with higher uncertainties, as these estimates exhibit greater tolerance to errors. As the combined depth network introduces increased complexity, we utilize Group Normalization (GN) to better capture spatial information in the prediction branch outputs. Furthermore, we leverage the Two-Dimensional (2D) information of objects to predict the residual in the 2D center after downsampling, aiding in the regression of 3D center. On the KITTI benchmark, our model achieves an average precision (AP) of 16.65 % and 23.19 % on 3D and bird's-eye view (BEV) detection for the moderate category, surpassing the state-of-the-art (SOTA) models in each category.

A Virtual View Acquisition Technique for Complex Scenes of Monocular Images Based on Layered Depth Images

With the rapid development of stereoscopic display technology, how to generate high-quality virtual view images has become the key in the applications of 3D video, 3D TV and virtual reality. The traditional virtual view rendering technology maps the reference view into the virtual view by means of 3D transformation, but when the background area is occluded by the foreground object, the content of the occluded area cannot be inferred. To solve this problem, we propose a virtual view acquisition technique for complex scenes of monocular images based on a layered depth image (LDI). Firstly, the depth discontinuities of the edge of the occluded area are reasonably grouped by using the multilayer representation of the LDI, and the depth edge of the occluded area is inpainted by the edge inpainting network. Then, the generative adversarial network (GAN) is used to fill the information of color and depth in the occluded area, and the inpainting virtual view is generated. Finally, GAN is used to optimize the color and depth of the virtual view, and the high-quality virtual view is generated. The effectiveness of the proposed method is proved by experiments, and it is also applicable to complex scenes.

Shape-from-Shading Derived DEMs from ShadowCam Images in Permanently Shadowed Regions at the Lunar South Pole: A First Look

Abstract. The lunar south pole, particularly its permanently shadowed regions (PSRs), has been the focus of future lunar exploration. The solar altitude angle at the lunar south pole is less than ∼ 2°, resulting in the PSRs not being directly illuminated and the possible existence of water ice in the PSRs due to the low temperatures. High-resolution digital elevation models (DEMs) in PSRs are crucial for future exploration missions and scientific research. However, existing image datasets (e.g., LROC NAC images) cannot penetrate the PSRs. Laser altimetry (e.g., LOLA) can measure the PSRs, but the DEMs derived from LOLA suffer from noises and interpolation defects. ShadowCam is a newly deployed camera onboard the Korea Pathfinder Lunar Orbiter, with a high imaging sensitivity. ShadowCam can capture the faint secondary illumination within PSRs, revealing previously indistinguishable topographic features in persistent darkness. Based on our existing Shape-from-Shading (SfS) pipeline for generating high-resolution DEMs from monocular images, we introduce a novel SfS approach that utilizes secondary illumination and reflectance to reconstruct pixel-wise high-resolution DEMs from ShadowCam images in PSRs. The approach consists of three steps: (1) modeling the secondary illumination on the target surfaces in PSRs based on the view factor; (2) optimizing a low-resolution LOLA DEM using the secondary illumination model; (3) refining the DEM to pixel-wise resolution based on SfS using the ShadowCam images. A test area (approximately 1 km × 1 km) within the PSR in the Shackleton crater near the lunar south pole is used for experimental evaluation. The results show that the DEM generated by the proposed approach is highly consistent with the direct LOLA measurements, particularly in small-scale topographic features such as crater floors and rims that are absent from the LOLA DEM. A first look at the results indicates that the proposed approach has great potential for generating high-resolution DEMs from ShadowCam images in PSRs, which can aid future exploration missions targeting the PSRs at the lunar south pole and support related scientific research on the water ice in the PSRs.

Spatio-temporal reconstruction of droplet impingement dynamics by means of color-coded glare points and deep learning

Abstract The present work introduces a deep learning approach for the three-dimensional reconstruction of the spatio-temporal dynamics of the gas–liquid interface on the basis of monocular images obtained via optical measurement techniques. The method is tested and evaluated at the example of liquid droplets impacting on structured solid substrates. The droplet dynamics are captured through high-speed imaging in an extended shadowgraphy setup with additional glare points from lateral light sources that encode further three-dimensional information of the gas–liquid interface in the images. A neural network is trained for the physically correct reconstruction of the droplet dynamics on a labeled dataset generated by synthetic image rendering on the basis of gas–liquid interface shapes obtained from direct numerical simulation. The employment of synthetic image rendering allows for the efficient generation of training data and circumvents the introduction of errors resulting from the inherent discrepancy of the droplet shapes between experiment and simulation. The accurate reconstruction of the three-dimensional shape of the gas–liquid interface during droplet impingement on the basis of images obtained in the experiment demonstrates the practicality of the presented approach based on neural networks and synthetic training data generation. The introduction of glare points from lateral light sources in the experiments is shown to improve the reconstruction accuracy, which indicates that the neural network learns to leverage the additional three-dimensional information encoded in the images for a more accurate depth estimation. By the successful reconstruction of obscured areas in the input images, it is demonstrated that the neural network has the capability to learn a physically correct interpolation of missing data from the numerical simulation. Furthermore, the physically reasonable reconstruction of unknown gas–liquid interface shapes for drop impact regimes that were not contained in the training dataset indicates that the neural network learned a versatile model of the involved two-phase flow phenomena during droplet impingement.

Exploring multi-scale forgery clues for stereo super-resolution image forgery localization

Existing forgery image localization methods have achieved impressive performance on monocular images but struggle to maintain comparable performance on stereo super-resolution (SR) images. The stereo image SR process usually introduces additional noise, i.e., reconstruction artifacts, which interferes with the extraction of forgery clues and thus undermines the localization performance. To mitigate this issue, we present a multi-scale attention framework named SSR-IFL for stereo SR image forgery localization, which explores multi-scale forgery clues in stereo SR images to minimize disturbances caused by reconstruction artifacts. Specifically, the novel framework integrates two significant components: the multi-scale feature extraction (MSFE) network to thoroughly mine forgery clues against the inference of reconstruction artifacts, and the bidirectional progressive feature fusion (BPFF) network to effectively reason the learned forgery clues and obtain robust forgery feature representation. Under the MSFE network, the multi-dilated self-attention (MDSA) block makes full use of multi-resolution pixel correlation to adapt to changes in pixel correlation distributions induced by the stereo SR reconstruction process, thus extracting more precise multi-scale forgery clues. The BPFF network introduces a bidirectional feature fusion strategy to fuse multi-scale forgery clues, effectively eliminating redundant noise such as reconstruction artifacts and enhancing the robustness of forgery feature representation. Experimental results on public image datasets demonstrate that our method can achieve not only competitive performance on stereo SR images but also on monocular images.

Innovative AI techniques for photorealistic 3D clothed human reconstruction from monocular images or videos: a survey

Innovative AI techniques for photorealistic 3D clothed human reconstruction from monocular images or videos: a survey

Learning Domain Invariant Features for Unsupervised Indoor Depth Estimation Adaptation

Predicting depth maps from monocular images has made an impressive performance in the past years. However, most depth estimation methods are trained with paired image-depth map data or multi-view images (e.g., stereo pair and monocular sequence), which suffer from expensive annotation costs and poor transferability. Although unsupervised domain adaptation methods are introduced to mitigate the reliance on annotated data, rare works focus on the unsupervised cross-scenario indoor monocular depth estimation. In this article, we propose to study the generalization of depth estimation models across different indoor scenarios in an adversarial-based domain adaptation paradigm. Concretely, a domain discriminator is designed for discriminating the representation from source and target domains, while the feature extractor aims to confuse the domain discriminator by capturing domain-invariant features. Further, we reconstruct depth maps from latent representations with the supervision of labeled source data. As a result, the feature extractor learned features possess the merit of both domain-invariant and low source risk, and the depth estimator can deal with the domain shift between source and target domains. We conduct the cross-scenario and cross-dataset experiments on the ScanNet and NYU-Depth-v2 datasets to verify the effectiveness of our method and achieve impressive performance.

Deformable surface reconstruction via Riemannian metric preservation

Estimating the pose of an object from a monocular image is a fundamental inverse problem in computer vision. Due to its ill-posed nature, solving this problem requires incorporating deformation priors. In practice, many materials do not perceptibly shrink or extend when manipulated, constituting a reliable and well-known prior. Mathematically, this translates to the preservation of the Riemannian metric. Neural networks offer the perfect playground to solve the surface reconstruction problem as they can approximate surfaces with arbitrary precision and allow the computation of differential geometry quantities. This paper presents an approach for inferring continuous deformable surfaces from a sequence of images, which is benchmarked against several techniques and achieves state-of-the-art performance without the need for offline training. Being a method that performs per-frame optimization, our method can refine its estimates, contrary to those based on performing a single inference step. Despite enforcing differential geometry constraints at each update, our approach is the fastest of all the tested optimization-based methods.

Monocular Pose and Shape Reconstruction of Vehicles in UAV imagery using a Multi-task CNN

Estimating the pose and shape of vehicles from aerial images is an important, yet challenging task. While there are many existing approaches that use stereo images from street-level perspectives to reconstruct objects in 3D, the majority of aerial configurations used for purposes like traffic surveillance are limited to monocular images. Addressing this challenge, a Convolutional Neural Network-based method is presented in this paper, which jointly performs detection, pose, type and 3D shape estimation for vehicles observed in monocular UAV imagery. For this purpose, a robust 3D object model is used following the concept of an Active Shape Model. In addition, different variants of loss functions for learning 3D shape estimation are presented, focusing on the height component, which is particularly challenging to estimate from monocular near-nadir images. We also introduce a UAV-based dataset to evaluate our model in addition to an augmented version of the publicly available Hessigheim benchmark dataset. Our method yields promising results in pose and shape estimation: utilising images with a ground sampling distance (GSD) of 3 cm, it achieves median errors of up to 4 cm in position and 3° in orientation. Additionally, it achieves root mean square (RMS) errors of \\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\pm 6$$\\end{document} cm in planimetry and \\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\pm 18$$\\end{document} cm in height for keypoints defining the car shape.

Prior-free 3D human pose estimation in a video using limb-vectors

Estimating accurate 3D human poses from a monocular video is fundamental to various computer vision tasks. Existing methods exploit 2D-to-3D pose lifting, multiview images, and depth sensors to model spatio-temporal dependencies. However, depth ambiguities, occlusions, and larger temporal receptive fields pose challenges to these approaches. To address this, we propose a novel prior-free DCNN-based 3D human pose estimation method for monocular image sequences using limb vectors. Our method comprises two subnetworks: a limb direction estimator and a limb length estimator. The limb direction estimator utilizes a fully convolutional network to model limb direction vectors across a temporal window. We show that network complexity can be significantly reduced by utilizing dilated convolutional operations and a relatively smaller receptive field while maintaining estimation accuracy. Moreover, the limb length estimator captures stable limb length estimations from a reliable frame set. Our model has shown superior performance compared to existing methods on the Human3.6M and MPI-INF-3DHP datasets.

Monocular BEV Perception of Road Scenes via Front-to-Top View Projection.

HD map reconstruction is crucial for autonomous driving. LiDAR-based methods are limited due to expensive sensors and time-consuming computation. Camera-based methods usually need to perform road segmentation and view transformation separately, which often causes distortion and missing content. To push the limits of the technology, we present a novel framework that reconstructs a local map formed by road layout and vehicle occupancy in the bird's-eye view given a front-view monocular image only. We propose a front-to-top view projection (FTVP) module, which takes the constraint of cycle consistency between views into account and makes full use of their correlation to strengthen the view transformation and scene understanding. In addition, we apply multi-scale FTVP modules to propagate the rich spatial information of low-level features to mitigate spatial deviation of the predicted object location. Experiments on public benchmarks show that our method achieves various tasks on road layout estimation, vehicle occupancy estimation, and multi-class semantic estimation, at a performance level comparable to the state-of-the-arts, while maintaining superior efficiency.

Learning accurate monocular 3D voxel representation via bilateral voxel transformer

Vision-based methods for 3D scene perception have been widely explored for autonomous vehicles. However, inferring complete 3D semantic scenes from monocular 2D images is still challenging owing to the 2D-to-3D transformation. Specifically, existing methods that use Inverse Perspective Mapping (IPM) to project image features to dense 3D voxels severely suffer from the ambiguous projection problem. In this research, we present Bilateral Voxel Transformer (BVT), a novel and effective Transformer-based approach for monocular 3D semantic scene completion. BVT exploits a bilateral architecture composed of two branches for preserving the high-resolution 3D voxel representation while aggregating contexts through the proposed Tri-Axial Transformer simultaneously. To alleviate the ill-posed 2D-to-3D transformation, we adopt position-aware voxel queries and dynamically update the voxels with image features through weighted geometry-aware sampling. BVT achieves 11.8 mIoU on the challenging Semantic KITTI dataset, considerably outperforming previous works for semantic scene completion with monocular images. The code and models of BVT will be available on GitHub.

LightDepth: A resource efficient depth estimation approach for dealing with ground truth sparsity via curriculum learning

Accurate depth estimation from monocular images is critical for various applications such as robotics, augmented reality, and autonomous navigation. However, achieving high accuracy while maintaining computational efficiency is a major challenge, particularly for resource-constrained devices. In this paper, we present LightDepth, an approach that leverages curriculum learning to estimate depth efficiently while taking into account resource constraints. It modifies the ground truth sparse depth maps from the KITTI dataset by resizing them to 31 extents during training to reduce sparsity and control complexity. The resulting model achieves comparable accuracy to state-of-the-art large models while outperforming them in response time by 71%. Our approach outperforms resource-efficient models regarding depth accuracy (measured by RMSE), achieving a 56% improvement. LightDepth is designed to be fast and resource-efficient, making it suitable for deployment in resource-constrained devices. It also balances the trade-off between accuracy and resource efficiency. All codes are available online at https://github.com/fatemehkarimii/lightdepth.

Shadow-constrained shape-from-shading for pixel-wise 3D surface reconstruction at the lunar south pole

ABSTRACT High-resolution digital elevation models (DEMs) of the lunar surface are crucial for lunar exploration missions and scientific research. The emergence of advanced technologies, such as Shape-from-Shading (SfS), has enabled the generation of pixel-wise high-resolution DEMs from monocular images of the lunar surface. However, SfS encounters significant challenges in locations with limited illumination, such as the lunar south pole, where the surface is largely covered by shadows. Therefore, this paper presents a novel shadow-constrained SfS approach for pixel-wise 3D surface reconstruction at the lunar south pole. The proposed approach uses multiple high-resolution images captured under different illumination conditions and an existing low-resolution DEM as the inputs and generates a high-resolution DEM with the same resolution as that of the input image through hierarchical SfS processing incorporating shadow constraints. Experiments were conducted using actual images collected by the Lunar Reconnaissance Orbiter (LRO) Narrow Angle Camera (NAC) at the lunar south pole. Comparisons with respect to photogrammetric DEMs generated from stereo NAC images show that the DEMs generated using the proposed approach exhibit the smallest root mean square error (RMSE). Moreover, shaded relief images rendered from the DEMs generated using the proposed approach demonstrate the highest similarity to the actual NAC images. Detailed profile comparisons further validate the effectiveness of the shadow constraint in optimizing 3D reconstruction in regions in proximity to shadows and within shadowed regions. The proposed shadow-constrained SfS approach can be used to generate high-resolution DEMs to support future missions to explore the lunar south pole, with applications including landing site evaluation and route planning for lunar probes or astronauts.

Artificial Intelligence for Early Detection of Pediatric Eye Diseases Using Mobile Photos

Identifying pediatric eye diseases at an early stage is a worldwide issue. Traditional screening procedures depend on hospitals and ophthalmologists, which are expensive and time-consuming. Using artificial intelligence (AI) to assess children's eye conditions from mobile photographs could facilitate convenient and early identification of eye disorders in a home setting. To develop an AI model to identify myopia, strabismus, and ptosis using mobile photographs. This cross-sectional study was conducted at the Department of Ophthalmology of Shanghai Ninth People's Hospital from October 1, 2022, to September 30, 2023, and included children who were diagnosed with myopia, strabismus, or ptosis. A deep learning-based model was developed to identify myopia, strabismus, and ptosis. The performance of the model was assessed using sensitivity, specificity, accuracy, the area under the curve (AUC), positive predictive values (PPV), negative predictive values (NPV), positive likelihood ratios (P-LR), negative likelihood ratios (N-LR), and the F1-score. GradCAM++ was utilized to visually and analytically assess the impact of each region on the model. A sex subgroup analysis and an age subgroup analysis were performed to validate the model's generalizability. A total of 1419 images obtained from 476 patients (225 female [47.27%]; 299 [62.82%] aged between 6 and 12 years) were used to build the model. Among them, 946 monocular images were used to identify myopia and ptosis, and 473 binocular images were used to identify strabismus. The model demonstrated good sensitivity in detecting myopia (0.84 [95% CI, 0.82-0.87]), strabismus (0.73 [95% CI, 0.70-0.77]), and ptosis (0.85 [95% CI, 0.82-0.87]). The model showed comparable performance in identifying eye disorders in both female and male children during sex subgroup analysis. There were differences in identifying eye disorders among different age subgroups. In this cross-sectional study, the AI model demonstrated strong performance in accurately identifying myopia, strabismus, and ptosis using only smartphone images. These results suggest that such a model could facilitate the early detection of pediatric eye diseases in a convenient manner at home.

Coplane-constrained sparse depth sampling and local depth propagation for depth estimation

Depth estimation with sparse reference has emerged recently, and predicts depth map from a monocular image and a set of depth reference samples. Previous works randomly select reference samples by sensors, leading to severe depth bias as this sampling is independent to image semantic and neglects the unbalance of depth distribution in regions. This paper proposes a Coplane-Constrained sparse Depth (CCD) sampling to explore representative reference samples, and design a Local Depth Propagation (LDP) network for complete the sparse point cloud map. This can capture diverse depth information and diffuse the valid points to neighbors with geometry prior. Specifically, we first construct the surface normal map and detect coplane pixels by superpixel segmenting for sampling references, whose depth can be represented by that of superpixel centroid. Then, we introduce local depth propagation to obtain coarse-level depth map with geometric information, which dynamically diffuses the depth from the reference to neighbors based on local planar assumption. Further, we generate the fine-level depth map by devising a pixel-wise focal loss, which imposes the semantic and geometry calibration on pixels with low confidence in coarse-level prediction. Extensive experiments on public datasets demonstrate that our model outperforms SOTA depth estimation and completion methods.

A Transformer‐Based Network for Full Object Pose Estimation with Depth Refinement

In response to increasing demand for robotics manipulation, accurate vision‐based full pose estimation is essential. While convolutional neural networks‐based approaches have been introduced, the quest for higher performance continues, especially for precise robotics manipulation, including in the Agri‐robotics domain. This article proposes an improved transformer‐based pipeline for full pose estimation, incorporating a Depth Refinement Module. Operating solely on monocular images, the architecture features an innovative Lighter Depth Estimation Network using a Feature Pyramid with an up‐sampling method for depth prediction. A Transformer‐based Detection Network with additional prediction heads is employed to directly regress object centers and predict the full poses of the target objects. A novel Depth Refinement Module is then utilized alongside the predicted centers, full poses, and depth patches to refine the accuracy of the estimated poses. The performance of this pipeline is extensively compared with other state‐of‐the‐art methods, and the results are analyzed for fruit picking applications. The results demonstrate that the pipeline improves the accuracy of pose estimation to up to 90.79% compared to other methods available in the literature.

Deep Learning-based Depth Estimation Methods from Monocular Image and Videos: A Comprehensive Survey

Estimating depth from single RGB images and videos is of widespread interest due to its applications in many areas, including autonomous driving, 3D reconstruction, digital entertainment, and robotics. More than 500 deep learning-based papers have been published in the past 10 years, which indicates the growing interest in the task. This paper presents a comprehensive survey of the existing deep learning-based methods, the challenges they address, and how they have evolved in their architecture and supervision methods. It provides a taxonomy for classifying the current work based on their input and output modalities, network architectures, and learning methods. It also discusses the major milestones in the history of monocular depth estimation, and different pipelines, datasets, and evaluation metrics used in existing methods.

A novel concurrent learning-based fixed-time convergent visual depth observer for weakly persistently exciting perspective dynamical systems

The problem of synthesizing a fixed-time depth observer based on monocular image feedback is considered in this study. The key challenge lies in the fact that the perspective dynamical system used to model visual motion is typically characterized as being weakly persistently exciting, which complicates observer synthesis. Furthermore, the key to achieving the task objective (safe obstacle avoidance, for instance) lies in the synthesis of a depth observer that achieves rapid convergence of the (static) obstacle depth estimate to the ground truth in a known fixed-time. To address these challenges, and in contrast with prior schemes that rely on a motion-restrictive persistency of excitation (PE) condition for ensuring exponential convergence, a novel adaptive observer framework is considered in this study that incorporates a concurrent learning (CL) term for ensuring fixed-time observer convergence. In particular, the use of concurrent learning allows for the synthesis of a relaxed finite-time excitation condition that relies on historical data recorded over a dynamic sliding window in the recent past that the proposed observer relies on to ensure fixed-time convergence. Thus, a continuous-time reduced-order observer formulation is presented that relies on camera motion data to achieve fixed-time convergence to a uniform ultimate bound for a suitably large choice of the observer gains. Experimental results are used to demonstrate the efficacy of the proposed scheme in the presence of significant measurement noise. A performance comparison study is also undertaken to demonstrate superior performance of the proposed scheme compared to leading alternative designs. Finally, the practical applicability of the proposed scheme is verified by incorporating the proposed scheme within a reactive navigation scheme to accomplish obstacle avoidance. By incorporating a suitably informative CL term within the observer framework, the proposed scheme eliminates the need to rely on a difficult-to-verify PE condition, thus rendering it more suitable for practical applications like visual target-tracking and visual servo control.