Single-view Image Research Articles

A Framework for Fall Detection Based on OpenPose Skeleton and LSTM/GRU Models

Falling is one of the causes of accidental death of elderly people over 65 years old in Taiwan. If the fall incidents are not detected in a timely manner, it could lead to serious injury or even death of those who fell. General fall detection approaches require the users to wear sensors, which could be cumbersome for the users to put on, and misalignment of sensors could lead to erroneous readings. In this paper, we propose using computer vision and applied machine-learning algorithms to detect fall without any sensors. We applied OpenPose real-time multi-person 2D pose estimation to detect movement of a subject using two datasets of 570 × 30 frames recorded in five different rooms from eight different viewing angles. The system retrieves the locations of 25 joint points of the human body and detects human movement through detecting the joint point location changes. The system is able to effectively identify the joints of the human body as well as filtering ambient environmental noise for an improved accuracy. The use of joint points instead of images improves the training time effectively as well as eliminating the effects of traditional image-based approaches such as blurriness, light, and shadows. This paper uses single-view images to reduce equipment costs. We experimented with time series recurrent neural network, long- and short-term memory, and gated recurrent unit models to learn the changes in human joint points in continuous time. The experimental results show that the fall detection accuracy of the proposed model is 98.2%, which outperforms the baseline 88.9% with 9.3% improvement.

Leveraging Image Analysis to Compute 3D Plant Phenotypes Based on Voxel-Grid Plant Reconstruction.

High throughput image-based plant phenotyping facilitates the extraction of morphological and biophysical traits of a large number of plants non-invasively in a relatively short time. It facilitates the computation of advanced phenotypes by considering the plant as a single object (holistic phenotypes) or its components, i.e., leaves and the stem (component phenotypes). The architectural complexity of plants increases over time due to variations in self-occlusions and phyllotaxy, i.e., arrangements of leaves around the stem. One of the central challenges to computing phenotypes from 2-dimensional (2D) single view images of plants, especially at the advanced vegetative stage in presence of self-occluding leaves, is that the information captured in 2D images is incomplete, and hence, the computed phenotypes are inaccurate. We introduce a novel algorithm to compute 3-dimensional (3D) plant phenotypes from multiview images using voxel-grid reconstruction of the plant (3DPhenoMV). The paper also presents a novel method to reliably detect and separate the individual leaves and the stem from the 3D voxel-grid of the plant using voxel overlapping consistency check and point cloud clustering techniques. To evaluate the performance of the proposed algorithm, we introduce the University of Nebraska-Lincoln 3D Plant Phenotyping Dataset (UNL-3DPPD). A generic taxonomy of 3D image-based plant phenotypes are also presented to promote 3D plant phenotyping research. A subset of these phenotypes are computed using computer vision algorithms with discussion of their significance in the context of plant science. The central contributions of the paper are (a) an algorithm for 3D voxel-grid reconstruction of maize plants at the advanced vegetative stages using images from multiple 2D views; (b) a generic taxonomy of 3D image-based plant phenotypes and a public benchmark dataset, i.e., UNL-3DPPD, to promote the development of 3D image-based plant phenotyping research; and (c) novel voxel overlapping consistency check and point cloud clustering techniques to detect and isolate individual leaves and stem of the maize plants to compute the component phenotypes. Detailed experimental analyses demonstrate the efficacy of the proposed method, and also show the potential of 3D phenotypes to explain the morphological characteristics of plants regulated by genetic and environmental interactions.

Depth estimation for underwater images from single view image

Underwater images often undergo distortions from scattering, absorption, colour loss, diffraction, polarisation, and varying attenuation depending on the light frequency, due to the water medium. This study answers problems associated with the recovery of underwater images, by developing novel depth map estimation which may be used as an intermediary step for underwater image restoration. The depth map is an important factor for the recovery of the underwater image, as it has been shown that proper estimation of depth, results in better restoration of the underwater image. The proposed algorithm estimates a depth map from a single view image, using blurriness and lighting information, obtained using a simple background neutralisation method. Results from the algorithm on selected raw underwater image data set have been compared with other algorithms and it has been shown that the proposed method gives Pearson coefficients in the range of 0.75–0.95, which are considerably higher and better log root mean square value compared to other methods in the literature. This suggests a better estimation of depth by the proposed method.

Multiview Feature Aggregation for Facade Parsing

Facade image parsing is essential to the semantic understanding and 3-D reconstruction of urban scenes. Considering the occlusion and appearance ambiguity in single-view images and the easy acquisition of multiple views, in this letter, we propose a multiview enhanced deep architecture for facade parsing. The highlight of this architecture is a cross-view feature aggregation module that can learn to choose and fuse useful convolutional neural network (CNN) features from nearby views to enhance the representation of a target view. Benefitting from the multiview enhanced representation, the proposed architecture can better deal with the ambiguity and occlusion issues. Moreover, our cross-view feature aggregation module can be straightforwardly integrated into existing single-image parsing frameworks. Extensive comparison experiments and ablation studies are conducted to demonstrate the good performance of the proposed method and the validity and transportability of the cross-view feature aggregation module.

Learning pose-invariant 3D object reconstruction from single-view images

Learning to reconstruct 3D shapes using 2D images is an active research topic, with benefits of not requiring expensive 3D data. However, most work in this direction requires multi-view images for each object instance as training supervision, which oftentimes does not apply in practice. In this paper, we relax the common multi-view assumption and explore a more challenging yet more realistic setup of learning 3D shape from only single-view images. The major difficulty lies in insufficient constraints that can be provided by single view images, which leads to the problem of pose entanglement in learned shape space. As a result, reconstructed shapes vary along input pose and have poor accuracy. We address this problem by taking a novel domain adaptation perspective, and propose an effective adversarial domain confusion method to learn pose-disentangled compact shape space. Experiments on single-view reconstruction show effectiveness in solving pose entanglement, and the proposed method achieves on-par reconstruction accuracy with state-of-the-art with higher efficiency.

Three-view generation based on a single front view image for car

The multi-view of an object can be used for 3D reconstruction. The method proposed in this paper generates the left and the top view of a target car through deep learning. The input of the method is only a front view of a 3D car and it isn’t necessary for the depth of the 3D car. Firstly, a rough orthographic views of the 3D car is gotten from an information constraint network which is constructed by considering the structural relation between one view and the other two views. And then the rough orthographic views is transformed into large-pixel block rough view through the nearest interpolation, at the same time, the large-pixel blocks are also migrated to improve the quality of the rough orthographic views. Finally, the generative adversarial network with perception loss is used to enhance the large-pixel block view. In addition, the three views generated by the network can be used to synthesize a 3D point cloud shell.

Differential Viewpoints for Ground Terrain Material Recognition.

Computational surface modeling that underlies material recognition has transitioned from reflectance modeling using in-lab controlled radiometric measurements to image-based representations based on internet-mined single-view images captured in the scene. We take a middle-ground approach for material recognition that takes advantage of both rich radiometric cues and flexible image capture. A key concept is differential angular imaging, where small angular variations in image capture enables angular-gradient features for an enhanced appearance representation that improves recognition. We build a large-scale material database, Ground Terrain in Outdoor Scenes (GTOS) database, to support ground terrain recognition for applications such as autonomous driving and robot navigation. The database consists of over 30,000 images covering 40 classes of outdoor ground terrain under varying weather and lighting conditions. We develop a novel approach for material recognition called texture-encoded angular network (TEAN) that combines deep encoding pooling of RGB information and differential angular images for angular-gradient features to fully leverage this large dataset. With this novel network architecture, we extract characteristics of materials encoded in the angular and spatial gradients of their appearance. Our results show that TEAN achieves recognition performance that surpasses single view performance and standard (non-differential/large-angle sampling) multiview performance.

Multi-View Pose Generator Based on Deep Learning for Monocular 3D Human Pose Estimation

In this paper, we study the problem of monocular 3D human pose estimation based on deep learning. Due to single view limitations, the monocular human pose estimation cannot avoid the inherent occlusion problem. The common methods use the multi-view based 3D pose estimation method to solve this problem. However, single-view images cannot be used directly in multi-view methods, which greatly limits practical applications. To address the above-mentioned issues, we propose a novel end-to-end 3D pose estimation network for monocular 3D human pose estimation. First, we propose a multi-view pose generator to predict multi-view 2D poses from the 2D poses in a single view. Secondly, we propose a simple but effective data augmentation method for generating multi-view 2D pose annotations, on account of the existing datasets (e.g., Human3.6M, etc.) not containing a large number of 2D pose annotations in different views. Thirdly, we employ graph convolutional network to infer a 3D pose from multi-view 2D poses. From experiments conducted on public datasets, the results have verified the effectiveness of our method. Furthermore, the ablation studies show that our method improved the performance of existing 3D pose estimation networks.

3D-Aware Scene Change Captioning From Multiview Images

In this letter, we propose a framework that recognizes and describes changes that occur in a scene observed from multiple viewpoints in natural language text. The ability to recognize and describe changes that occurred in a 3D scene plays an essential role in a variety of human-robot interaction applications. However, most current 3D vision studies have focused on understanding the static 3D scene. Existing scene change captioning approaches recognize and generate change captions from single-view images. Those methods have limited ability to deal with camera movement, object occlusion, which are common in real-world settings. To resolve these problems, we propose a framework that observes every scene from multiple viewpoints and describes the scene change based on an understanding of the underlying 3D structure of scenes. We build three synthetic datasets consisting of primitive 3D object and scanned real object models for evaluation. The results indicate that our method outperforms the previous state-of-the-art 2D-based method by a large margin in terms of sentence generation and change understanding correctness. In addition, our method is more robust to camera movements compared to the previous method and also performs better for scenes with occlusions. Moreover, our method also shows encouraging results in a realistic scene-setting, which indicates the possibility of adapting our framework to a more complicated and extensive scene-settings.

Anthropometric Parameter Measurement from Equivariant Multi-view Images

In this paper, we propose an anthropometric parameter measurement method that any customized parameter can be measured online by the pre-selected endpoints on the reconstructed 3D body models of equivariant multi-view images. The method includes 3D body model reconstruction, anthropometric parameter measurement, and parameter modification. In 3D body model reconstruction, we detect and segment the human body from its background and reconstruct a generative 3D body model from the segmented image with deep learning. And then we measure anthropometric parameter on the reconstructed 3D body model of each view. Before parameter measurement, we manually pre-select endpoints associated with all anthropometric parameters on the reconstructed 3D body model since all vertices of the reconstructed body model are ordered. However, the information of a single-view image is insufficient and the measurement result is varied regularly by the view changes. To improve the measurement accuracy, we design a convolutional neural network in the last step which can regress more accurate anthropometric parameters from equivariant multi-view measurements. Experimental results on the representative dataset demonstrate that the proposed method can measure planar and spatial anthropometric parameters automatically with comparable performance.

Fast and Accurate Self-calibration Using Vanishing Point Detection in Manmade Environments

Interests of auto-calibration have been increased in several camera systems. This paper presents a novel self-calibration method using fast and accurate vanishing point detection algorithm that works in manmade environments. The proposed algorithm estimates focal length assuming that the principal point is the center of an image to satisfy the orthogonality of three vanishing points. By using proposed vanishing point detection algorithm and minimization of the proposed objective function, the proposed system detects accurate vanishing points with focal length outperforming other methods. The proposed vanishing point detection algorithm detects vanishing points by using J-linkage based method that is more delicate by fragmentation and re-merging strategies. The proposed objective function finally detects vanishing points that meets orthogonality among estimated hypotheses for vanishing points by checking several geometric relationships. We believe that the proposed method can be used for automatic camera calibration, localization of a camera in an autonomous navigation system, and three-dimensional reconstruction of a single-view image.

Multi-View Visual Question Answering with Active Viewpoint Selection.

This paper proposes a framework that allows the observation of a scene iteratively to answer a given question about the scene. Conventional visual question answering (VQA) methods are designed to answer given questions based on single-view images. However, in real-world applications, such as human–robot interaction (HRI), in which camera angles and occluded scenes must be considered, answering questions based on single-view images might be difficult. Since HRI applications make it possible to observe a scene from multiple viewpoints, it is reasonable to discuss the VQA task in multi-view settings. In addition, because it is usually challenging to observe a scene from arbitrary viewpoints, we designed a framework that allows the observation of a scene actively until the necessary scene information to answer a given question is obtained. The proposed framework achieves comparable performance to a state-of-the-art method in question answering and simultaneously decreases the number of required observation viewpoints by a significant margin. Additionally, we found our framework plausibly learned to choose better viewpoints for answering questions, lowering the required number of camera movements. Moreover, we built a multi-view VQA dataset based on real images. The proposed framework shows high accuracy (94.01%) for the unseen real image dataset.

PGNet: A Part-based Generative Network for 3D object reconstruction

Deep-learning generative methods have developed rapidly. For example, various single- and multi-view generative methods for meshes, voxels, and point clouds have been introduced. However, most 3D single-view reconstruction methods generate whole objects at one time, or in a cascaded way for dense structures, which misses local details of fine-grained structures. These methods are useless when the generative models are required to provide semantic information for parts. This paper proposes an efficient part-based recurrent generative network, which aims to generate object parts sequentially with the input of a single-view image and its semantic projection. The advantage of our method is its awareness of part structures; hence it generates more accurate models with fine-grained structures. Experiments show that our method attains high accuracy compared with other point set generation methods, particularly toward local details.

3D-ReConstnet: A Single-View 3D-Object Point Cloud Reconstruction Network

Object 3D reconstruction from a single-view image is an ill-posed problem. Inferring the self-occluded part of an object makes 3D reconstruction a challenging and ambiguous task. In this paper, we propose a novel neural network for generating a 3D-object point cloud model from a single-view image. The proposed network named 3D-ReConstnet, an end to end reconstruction network. The 3D-ReConstnet uses the residual network to extract the features of a 2D input image and gets a feature vector. To deal with the uncertainty of the self-occluded part of an object, the 3D-ReConstnet uses the Gaussian probability distribution learned from the feature vector to predict the point cloud. The 3D-ReConstnet can generate the determined 3D output for a 2D image with sufficient information, and 3D-ReConstnet can also generate semantically different 3D reconstructions for the self-occluded or ambiguous part of an object. We evaluated the proposed 3D-ReConstnet on ShapeNet and Pix3D dataset, and obtained satisfactory improved results.

Graph Convolutional Adversarial Network for Human Body Pose and Mesh Estimation

This paper studies reconstruction of human body shape and pose from a single-view image. While most of current work attempts to regress parameters of human body model such as Skinned Multi-Person Linear Model (SMPL) and Hand Model with Articulated and Non-rigid Deformations (MANO), these parametric approaches underperform compared to non-parametric approaches. Due to the lack of the spatial relationship in the input image, the parametric approaches are hardly used to reconstruct the human body precisely. Besides, the rotation parameter regression is a complex task in parametric approaches. Therefore, we introduce a novel graph convolutional neural network (Graph CNN)-based framework for estimating a non-parametric mesh model. Our key innovation is that the proposed model is trained in a generative adversarial manner. Firstly, Graph CNN utilizes mesh topology to capture integral information of the full 3D human shape and then generate a more smooth and high-quality human mesh model. Secondly, the discriminator in our network acts as a supervisor to specify whether a human shape and pose are real or not. The generator is encouraged to generate human body mesh that is close to the manifold of the real human mesh distribution. Extensive experimental results demonstrate the effectiveness of our proposed framework. In contrast to the state-of-the-art methods, our method can achieve better performance in human shape and pose estimation.

Semantic based autoencoder-attention 3D reconstruction network

3D object reconstruction from single view image is a challenge task. Due to the fact that the information contained in one isolated image is not sufficient for reasonable 3D shape reconstruction, the existing methods on single-view 3D reconstruction always lack marginal voxels and miss obvious semantic information. To tackle these problem, we propose Semantic Autoencoder-Attention Network (SAAN) for single view 3D reconstruction. Distinct from the common auto-encoder (AE) structure, the proposed network consists of two successive parts. The first part is made of two parallel branches, 3D-autoencoder (3DAE) and Attention Network. 3DAE completes the general shape reconstruction by an AE model, and Attention Network supplements the missing details by a 3D reconstruction attention network. In the other part, we compare the semantic information between the input images and the generated voxel from the autoencoder-attention network (AAN) with Res101 network and 3D-Res101 network. The results of the comparison are fed back to the AAN for parameter adjustment iteratively until to generate the best semantically characterized voxel representations. In the experiments, we verify the feasibility of our network on the ShapeNet dataset. By comparing with the state-of-art methods, the proposed SAAN can produce more precise 3D object models in terms of both qualitative and quantitative evaluation.

Depth Analysis of Single View Image Objects based on Object Detection and Focus Measure

Depth Analysis of Single View Image Objects based on Object Detection and Focus Measure

A De novo approach for automatic volume and mass estimation of fruits and vegetables

Fruit and vegetable quality grading is a major task for successful marketing in the agricultural industry. There are many criteria to determine the quality of fruits and vegetables out of which volume and mass estimation are important ones. The current processes of volume and mass estimation have some drawbacks while considered for a large number of fruits and vegetables in the agricultural industry. These drawbacks are the requirement of highly efficient manual resource and long time. The manual processing is sometimes destructive in nature for fresh fruits and vegetables. An automated and non-destructive technique based on image analysis may alleviate the problem to a certain extent. Computer vision-based solution became challenging for irregular shaped fruits and vegetables from a single view image. This paper addresses the challenges by using an innovative split and merge technique. The input image has been passed through a few pre-processing stages to reduce the noise as well as segment the object from the background. Then the segmented object has been split into two separate objects along the major axis. The valid boundary points are extracted and subsequently approximated by a polynomial expression for each part. The volume is extracted by integrating the polynomial. The volumes from the parts are summed to get the volume of entire fruit or vegetable. Mass is computed by using the density, mass and volume relationship.

Edge Detection Algorithm of a Symmetric Difference Kernel SAR Image Based on the GAN Network Model

The symmetrical difference kernel SAR image edge detection algorithm based on the Canny operator can usually achieve effective edge detection of a single view image. When detecting a multi-view SAR image edge, it has the disadvantage of a low detection accuracy. An edge detection algorithm for a symmetric difference nuclear SAR image based on the GAN network model is proposed. Multi-view data of a symmetric difference nuclear SAR image are generated by the GAN network model. According to the results of multi-view data generation, an edge detection model for an arbitrary direction symmetric difference nuclear SAR image is constructed. A non-edge is eliminated by edge post-processing. The Hough transform is used to calculate the edge direction to realize the accurate detection of the edge of the SAR image. The experimental results show that the average classification accuracy of the proposed algorithm is 93.8%, 96.85% of the detection edges coincide with the correct edges, and 97.08% of the detection edges fall into the buffer of three pixel widths, whichshows that the proposed algorithm has a high accuracy of edge detection for kernel SAR images.

Data-Driven Point Cloud Objects Completion.

With the development of the laser scanning technique, it is easier to obtain 3D large-scale scene rapidly. However, many scanned objects may suffer serious incompletion caused by the scanning angles or occlusion, which has severely impacted their future usage for the 3D perception and modeling, while traditional point cloud completion methods often fails to provide satisfactory results due to the large missing parts. In this paper, by utilising 2D single-view images to infer 3D structures, we propose a data-driven Point Cloud Completion Network (), which is an image-guided deep-learning-based object completion framework. With the input of incomplete point clouds and the corresponding scanned image, the network can acquire enough completion rules through an encoder-decoder architecture. Based on an attention-based 2D-3D fusion module, the network is able to integrate 2D and 3D features adaptively according to their information integrity. We also propose a projection loss as an additional supervisor to have a consistent spatial distribution from multi-view observations. To demonstrate the effectiveness, first, the proposed is compared to recent generative networks and has shown more powerful 3D reconstruction abilities. Then, is compared to a recent point cloud completion methods, which has demonstrate that the proposed is able to provide satisfied completion results for objects with large missing parts.