Single RGB Image Research Articles

Vehicle Lane Change Prediction on Highways Using Efficient Environment Representation and Deep Learning

This paper introduces a novel method of lane-change and lane-keeping detection and prediction of surrounding vehicles based on Convolutional Neural Network (CNN) classification approach. Context, interaction, vehicle trajectories, and scene appearance are efficiently combined into a single RGB image that is fed as input for the classification model. Several state-of-the-art classification-CNN models of varying complexity are evaluated to find out the most suitable one in terms of anticipation and prediction. The model has been trained and evaluated using the PREVENTION dataset, a specific dataset oriented to vehicle maneuver and trajectory prediction. The proposed model can be trained and used to detect lane changes as soon as they are observed, and to predict them before the lane change maneuver is initiated. Concurrently, a study on human performance in predicting lane-change maneuvers using visual inputs has been conducted, so as to establish a solid benchmark for comparison. The empirical study reveals that humans are able to detect the 83.9% of lane changes on average 1.66 seconds in advance. The proposed automated maneuver detection model increases anticipation by 0.43 seconds and accuracy by 2.5% compared to human results, while the maneuver prediction model increases anticipation by 1.03 seconds with an accuracy decrease of only 0.5%.

Noninvasive Diagnosis of Seedless Fruit Using Deep Learning in Persimmon

Noninvasive diagnosis of internal traits in fruit crops is a high unmet need; however it generally requires time, costs, and special methods or facilities. Recent progress in deep neural network (or deep learning) techniques would allow easy, but highly accurate diagnosis with single RGB images, and the latest applications enable visualization of “the reasons for each diagnosis” by backpropagation of neural networks. Here, we propose an application of deep learning for image diagnosis on the classification of internal fruit traits, in this case seedlessness, in persimmon fruit (Diospyros kaki). We examined the classification of seedlessness in persimmon fruit by using four convolutional neural networks (CNN) models with various layer structures. With only 599 pictures of ‘Fuyu’ persimmon fruit from the fruit apex side, the neural networks successfully made a binary classification of seedless and seeded fruits with up to 85% accuracy. Among the four CNN models, the VGG16 model with the simplest layer structure showed the highest classification accuracy of 89%. Prediction values for the binary classification of seeded fruits were significantly increased in proportion to seed numbers in all four CNN models. Furthermore, explainable AI methods, such as Gradient-weighted Class Activation Mapping (Grad-CAM) and Guided Grad-CAM, allowed visualization of the parts and patterns contributing to the diagnosis. The results indicated that finer positions surrounding the apex, which correspond to hypothetical bulges derived from seeds, are an index for seeded fruits. These results suggest the novel potential of deep learning for noninvasive diagnosis of fruit internal traits using simple RGB images and also provide novel insights into previously unrecognized features of seeded/seedless fruits.

Dilated U-Block for Lightweight Indoor Depth Completion With Sobel Edge

Depth completion has proven to be the key to obtaining high-precision depth maps from a sparse set of measurements and a single RGB image. However, state-of-the-art depth completion algorithms are based on rather complex deep networks that are unfriendly for multiple applications with limited computational resources and energy. In this paper, we consider the task of lightweight indoor depth completion. We propose an efficient convolution block, called Dilated U-Block (DUB), for multi-level feature extraction and integration. The DUBs and an auxiliary Sobel edge prediction network are used to decrease the number of parameters and model complexity. Comprehensive experiments are performed on the indoor NYU-Depth-v2 dataset. The results show that our proposed approach achieves similar accuracy while requiring only about 5% of parameter size and 20% model complexity compared to the state-of-the-art methods.

Detecting Object Surface Keypoints From a Single RGB Image via Deep Learning Network for 6-DoF Pose Estimation

Estimating the 6-DoF (Degree of Freedom) object pose from a single RGB image is one of the challenging tasks in the field of computer vision. Before the pose which is defined as the translation and rotation parameters can be derived by the traditional PnP algorithm, 2D image projections of a set of 3D object keypoints must be accurately detected. In this paper, we present techniques for defining 3D object surface keypoints and predicting their corresponding 2D counterparts via deep-learning network architectures. The main technique to designate 3D object keypoints is to employ quadratic fitting scheme for calculating the principal surface curvatures as the weights and then select from all surface points the ones mostly distributive with larger curvatures to describe the object shape as possible. However, the 2D projected keypoints are not directly regressed from the network, but encoded as the unit vector fields pointing to them, so that the voting scheme to recover back those 2D keypoints can be performed. Moreover, an effective loss function with the regularization term is adopted in training ResNet for predicting image projections of object keypoints by focusing on small-scale errors. Experimental results show that our proposed technique outperforms state-of-the-art approaches in both “2D projection” and “3D transformation” metrics.

Deep Amended Gradient Descent for Efficient Spectral Reconstruction From Single RGB Images

This paper investigates the problem of recovering hyperspectral (HS) images from single RGB images. To tackle such a severely ill-posed problem, we propose a physically-interpretable, compact, efficient, and end-to-end learning-based framework, namely AGD-Net. Precisely, by taking advantage of the imaging process, we first formulate the problem explicitly based on the classic gradient descent algorithm. Then, we design a lightweight neural network with a multi-stage architecture to mimic the formed amended gradient descent process, in which efficient convolution and novel spectral zero-mean normalization are proposed to effectively extract spatial-spectral features for regressing an initialization, a basic gradient, and an incremental gradient. Besides, based on the approximate low-rank property of HS images, we propose a novel rank loss to promote the similarity between the global structures of reconstructed and ground-truth HS images, which is optimized with our singular value weighting strategy during training. Moreover, AGD-Net, a single network after one-time training, is flexible to handle the reconstruction with various spectral response functions. Extensive experiments over three commonly-used benchmark datasets demonstrate that AGD-Net can improve the reconstruction quality by more than 1.0 dB on average while saving 67× parameters and 32× FLOPs, compared with state-of-the-art methods. The code will be publicly available at https://github.com/zbzhzhy/GD-Net .

Encoder-Decoder Structure With the Feature Pyramid for Depth Estimation From a Single Image

We address the problem of depth estimation from a single monocular image in the paper. Depth estimation from a single image is an ill-posed and inherently ambiguous problem. In the paper, we propose an encoder-decoder structure with the feature pyramid to predict the depth map from a single RGB image. More specifically, the feature pyramid is used to detect objects of different scales in the image. The encoder structure aims to extract the most representative information from the original image through a series of convolution operations and to reduce the resolution of the input image. We adopt Res2-50 as the encoder to extract important features. The decoder section uses a novel upsampling structure to improve the output resolution. Then, we also propose a novel loss function that adds gradient loss and surface normal loss to the depth loss, which can predict not only the global depth but also the depth of fuzzy edges and small objects. Additionally, we use Adam as our optimization function to optimize our network and speed up convergence. Our extensive experimental evaluation proves the efficiency and effectiveness of the method, which is competitive with previous methods on the Make3D dataset and outperforms state-of-the-art methods on the NYU Depth v2 dataset.

Adaptive Context-Aware Multi-Modal Network for Depth Completion

Depth completion aims to recover a dense depth map from the sparse depth data and the corresponding single RGB image. The observed pixels provide the significant guidance for the recovery of the unobserved pixels' depth. However, due to the sparsity of the depth data, the standard convolution operation, exploited by most of existing methods, is not effective to model the observed contexts with depth values. To address this issue, we propose to adopt the graph propagation to capture the observed spatial contexts. Specifically, we first construct multiple graphs at different scales from observed pixels. Since the graph structure varies from sample to sample, we then apply the attention mechanism on the propagation, which encourages the network to model the contextual information adaptively. Furthermore, considering the mutli-modality of input data, we exploit the graph propagation on the two modalities respectively to extract multi-modal representations. Finally, we introduce the symmetric gated fusion strategy to exploit the extracted multi-modal features effectively. The proposed strategy preserves the original information for one modality and also absorbs complementary information from the other through learning the adaptive gating weights. Our model, named Adaptive Context-Aware Multi-Modal Network (ACMNet), achieves the state-of-the-art performance on two benchmarks, i.e., KITTI and NYU-v2, and at the same time has fewer parameters than latest models. Our code is available at: https://github.com/sshan-zhao/ACMNet.

D-Net: A Generalised and Optimised Deep Network for Monocular Depth Estimation

Depth estimation is an essential component in computer vision systems for achieving 3D scene understanding. Efficient and accurate depth map estimation has numerous applications including self-driving vehicles and virtual reality tools. This paper presents a new deep network, called D-Net, for depth estimation from a single RGB image. The proposed network can be trained end-to-end, and its structure can be customised to meet different requirements in model size, speed, and prediction accuracy. Our approach gathers strong global and local contextual features at multiple resolutions, and then transfers these to high resolutions for clearer depth maps. For the encoder backbone, D-Net can utilise many state-of-the-art models including EfficientNet, HRNet and Swin Transformer to obtain dense depth maps. The proposed D-net is designed to have minimal parameters and reduced computational complexity. Extensive evaluations on the NYUv2 and KITTI benchmark datasets show that our model is highly accurate across multiple backbones, and it achieves state-of-the-art performance on both benchmarks when combined with the Swin Transformer and HRNets.

MGG: Monocular Global Geolocation for Outdoor Long-Range Targets.

Traditional monocular vision localization methods are usually suitable for short-range area and indoor relative positioning tasks. This paper presents MGG, a novel monocular global geolocation method for outdoor long-range targets. This method takes a single RGB image combined with necessary navigation parameters as input and outputs targets' GPS information under the Global Navigation Satellite System (GNSS). In MGG, we first design a camera pose correction method via pixel mapping to correct the pose of the camera. Then, we use anchor-based methods to improve the detection ability for long-range targets with small image regions. Next, the local monocular vision model (LMVM) with a local structure coefficient is proposed to establish an accurate 2D-to-3D mapping relationship. Subsequently, a soft correspondence constraint (SCC) is presented to solve the local structure coefficient, which can weaken the coupling degree between detection and localization. Finally, targets can be geolocated through optimization theory-based methods and a series of coordinate transformations. Furthermore, we demonstrate the importance of focal length on solving the error explosion problem in locating long-range targets with monocular vision. Extensive experiments on the challenging KITTI dataset as well as applications in outdoor environments with targets located at a long range of up to 150 meters show the superiority of our method.

3D object detection based on synthetic RGB image

With the rapid development of computer vision technology and the maturity of 2D object detection technology, 3D object detection has attracted more and more attention of scholars. As an extension of 2D object detection, 3D object detection can not only detect the object category and position in the image, but also output the spatial position and pose angle of the object in a certain 3D coordinate system. It has a considerable application prospects in automatic driving, robot operation and augmented reality. However, 3D object detection is much more difficult than 2D object detection, and it is very difficult to make and label the data set, which has high hardware requirements. At present, there is no dominant algorithm, and scholars use a variety of methods. In order to solve the above problems, this paper proposes a method of making and synthesising 3D object detection data sets. Based on the improved 2D object detection algorithm, 3D object detection of single RGB image is realised.

Improved Multiview Decomposition for Single-Image High-Resolution 3D Object Reconstruction

As a representative technology of artificial intelligence, 3D reconstruction based on deep learning can be integrated into the edge computing framework to form an intelligent edge and then realize the intelligent processing of the edge. Recently, high-resolution representation of 3D objects using multiview decomposition (MVD) architecture is a fast reconstruction method for generating objects with realistic details from a single RGB image. The results of high-resolution 3D object reconstruction are related to two aspects. On the one hand, a low-resolution reconstruction network represents a good 3D object from a single RGB image. On the other hand, a high-resolution reconstruction network maximizes fine low-resolution 3D objects. To improve these two aspects and further enhance the high-resolution reconstruction capabilities of the 3D object generation network, we study and improve the low-resolution 3D generation network and the depth map superresolution network. Eventually, we get an improved multiview decomposition (IMVD) network. First, we use a 2D image encoder with multifeature fusion (MFF) to enhance the feature extraction capability of the model. Second, a 3D decoder using an effective subpixel convolutional neural network (3D ESPCN) improves the decoding speed in the decoding stage. Moreover, we design a multiresidual dense block (MRDB) to optimize the depth map superresolution network, which allows the model to capture more object details and reduce the model parameters by approximately 25% when the number of network layers is doubled. The experimental results show that the proposed IMVD is better than the original MVD in the 3D object superresolution experiment and the high-resolution 3D reconstruction experiment of a single image.

PVNet: Pixel-Wise Voting Network for 6DoF Object Pose Estimation.

This paper addresses the problem of instance-level 6DoF object pose estimation from a single RGB image. Many recent works have shown that a two-stage approach, which first detects keypoints and then solves a Perspective-n-Point (PnP) problem for pose estimation, achieves remarkable performance. However, most of these methods only localize a set of sparse keypoints by regressing their image coordinates or heatmaps, which are sensitive to occlusion and truncation. Instead, we introduce a Pixel-wise Voting Network (PVNet) to regress pixel-wise vectors pointing to the keypoints and use these vectors to vote for keypoint locations. This creates a flexible representation for localizing occluded or truncated keypoints. Another important feature of this representation is that it provides uncertainties of keypoint locations that can be further leveraged by the PnP solver. Experiments show that the proposed approach outperforms the state of the art on the LINEMOD, Occluded LINEMOD, YCB-Video, and Tless datasets, while being efficient for real-time pose estimation. We further create a Truncated LINEMOD dataset to validate the robustness of our approach against truncation. The code is available at https://github.com/zju3dv/pvnet.

Single View 3D Reconstruction Based on Improved RGB-D Image

Image-based 3D reconstruction is a fundamental task in computer vision and computer graphics. Although many existing approaches have obtained excellent results, they all dependent on multi-view images, then these methods are very complex. To reduce the complexity of the existing approaches, in this paper we fist propose a novel 3D reconstruction method based on single RGB image and the corresponding depth image, and design a unified framework called Single3D. Second, we trained a neural network model to reduce noises for improving the quality of the input RGB image, then leading a desirable RGB image. Third, we combine the deep guided filter and fast guided filter to fill holes on the depth image for improving the quality of 3D model. Fourth, we recover high-quality 3D model from the RGB-D image based on the proposed Single3D framework. Finally, the comprehensive experiments were conducted on the most challenging benchmarking dataset with variant light, occlusion, and clustering. Experimental results show that our method has desirable accuracy and efficiency.

Deep Learning in Hyperspectral Image Reconstruction from Single RGB images—A Case Study on Tomato Quality Parameters

Hyperspectral imaging has many applications. However, the high device costs and low hyperspectral image resolution are major obstacles limiting its wider application in agriculture and other fields. Hyperspectral image reconstruction from a single RGB image fully addresses these two problems. The robust HSCNN-R model with mean relative absolute error loss function and evaluated by the Mean Relative Absolute Error metric was selected through permutation tests from models with combinations of loss functions and evaluation metrics, using tomato as a case study. Hyperspectral images were subsequently reconstructed from single tomato RGB images taken by a smartphone camera. The reconstructed images were used to predict tomato quality properties such as the ratio of soluble solid content to total titratable acidity and normalized anthocyanin index. Both predicted parameters showed very good agreement with corresponding “ground truth” values and high significance in an F test. This study showed the suitability of hyperspectral image reconstruction from single RGB images for fruit quality control purposes, underpinning the potential of the technology—recovering hyperspectral properties in high resolution—for real-world, real time monitoring applications in agriculture any beyond.

Deep-learning-based hyperspectral recovery from a single RGB image.

Commercial hyperspectral imaging devices are expensive and tend to suffer from the degradation of spatial, spectral, or temporal resolution. To address these problems, we propose a deep-learning-based method to recover hyperspectral images from a single RGB image. The proposed method learns an end-to-end mapping between an RGB image and corresponding hyperspectral images. Moreover, a customized loss function is proposed to boost the performance. Experimental results on a variety of hyperspectral datasets demonstrate that our proposed method outperforms several state-of-the-art methods in terms of both quantitative measurements and perceptual quality.

Multi‐scale Information Assembly for Image Matting

AbstractImage matting is a long‐standing problem in computer graphics and vision, mostly identified as the accurate estimation of the foreground in input images. We argue that the foreground objects can be represented by different‐level information, including the central bodies, large‐grained boundaries, refined details, etc. Based on this observation, in this paper, we propose a multi‐scale information assembly framework (MSIA‐matte) to pull out high‐quality alpha mattes from single RGB images. Technically speaking, given an input image, we extract advanced semantics as our subject content and retain initial CNN features to encode different‐level foreground expression, then combine them by our well‐designed information assembly strategy. Extensive experiments can prove the effectiveness of the proposed MSIA‐matte, and we can achieve state‐of‐the‐art performance compared to most existing matting networks.

Real-time 3D Perception of Scene with Monocular Camera


 
 
 Depth is a vital prerequisite for the fulfillment of various tasks such as perception, navigation, and planning. Estimating depth using only a single image is a challenging task since the analytic mapping is not available between the intensity image and its depth where the features cue of the context is usually absent in the single image. Furthermore, most current researchers rely on the supervised Learning approach to handle depth estimation. Therefore, the demand for recorded ground truth depth is important at the training time, which is actually tricky and costly. This study presents two approaches (unsupervised learning and semi-supervised learning) to learn the depth information using only a single RGB-image. The main objective of depth estimation is to extract a representation of the spatial structure of the environment and to restore the 3D shape and visual appearance of objects in imagery.
 
 

Self-supervised pain intensity estimation from facial videos via statistical spatiotemporal distillation

• We propose a self-supervised representation learning for pain intensity estimation from the face. • We use a novel similarity function in a Siamese network to learn representations from unlabeled data as the pretext task. • Statistical Spatiotemporal Distillation is presented to encode dynamic and appearance information of video into an image map. • Our proposed method shows promising performance in cross-dataset evaluation setting using two publicly available datasets. Recently, automatic pain assessment technology, in particular automatically detecting pain from facial expressions, has been developed to improve the quality of pain management, and has attracted increasing attention. In this paper, we propose self-supervised learning for automatic yet efficient pain assessment, in order to reduce the cost of collecting large amount of labeled data. To achieve this, we introduce a novel similarity function to learn generalized representations using a Siamese network in the pretext task. The learned representations are finetuned in the downstream task of pain intensity estimation. To make the method computationally efficient, we propose Statistical Spatiotemporal Distillation (SSD) to encode the spatiotemporal variations underlying the facial video into a single RGB image, enabling the use of less complex 2D deep models for video representation. Experiments on two publicly available pain datasets and cross-dataset evaluation demonstrate promising results, showing the good generalization ability of the learned representations.

Disentangling Monocular 3D Object Detection: From Single to Multi-Class Recognition.

In this paper we introduce a method for multi-class, monocular 3D object detection from a single RGB image, which exploits a novel disentangling transformation and a novel, self-supervised confidence estimation method for predicted 3D bounding boxes. The proposed disentangling transformation isolates the contribution made by different groups of parameters to a given loss, without changing its nature. This brings two advantages: i) it simplifies the training dynamics in the presence of losses with complex interactions of parameters; and ii) it allows us to avoid the issue of balancing independent regression terms. We further apply this disentangling transformation to another novel, signed Intersection-over-Union criterion-driven loss for improving 2D detection results. We also critically review the AP metric used in KITTI3D and resolve a flaw which affected and biased all previously published results on monocular 3D detection. Our improved metric is now used as official KITTI3D metric. We provide extensive experimental evaluations and ablation studies on the KITTI3D and nuScenes datasets, setting new state-of-the-art results. We provide additional results on all the classes of KITTI3D as well as nuScenes datasets to further validate the robustness of our method, demonstrating its ability to generalize for different types of objects.

Consistent-Resolution Network for 3D Hand Shape Estimation from a Single RGB Image

We propose a novel method for 3D hand shape estimation from a single RGB image. Most exiting methods leverage a deep network to extract a low-resolution representation to estimate 3D coordinates, which always leads to the loss of spatial information. In contrast, we present a Consistent-Resolution Network (CRNet) to extract the same resolution representation as the original image, thus preserve more details about spatial information. Specifically, we introduce the recent high-resolution network (HRNet) to generate high-resolution feature maps, which can attain high-resolution representation of the original image. Then, we design a deconvolution module to recover this map to the size of the original image. Therefore, we can directly leverage this feature to learn the precise 2D shape and the depth map, and transfer them into 3D coordinates in the camera space. Through extensive experiments on a large real-world dataset FreiHAND, we show that our proposed method can predict precise and suitable 3D hand shape from a monocular view.