Natural Outdoor Scenes Research Articles

CNN-Based Object Detection via Segmentation Capabilities in Outdoor Natural Scenes

CNN-Based Object Detection via Segmentation Capabilities in Outdoor Natural Scenes

An Active Multi-Object Ultrafast Tracking System with CNN-Based Hybrid Object Detection.

This study proposes a visual tracking system that can detect and track multiple fast-moving appearance-varying targets simultaneously with 500 fps image processing. The system comprises a high-speed camera and a pan-tilt galvanometer system, which can rapidly generate large-scale high-definition images of the wide monitored area. We developed a CNN-based hybrid tracking algorithm that can robustly track multiple high-speed moving objects simultaneously. Experimental results demonstrate that our system can track up to three moving objects with velocities lower than 30 m per second simultaneously within an 8-m range. The effectiveness of our system was demonstrated through several experiments conducted on simultaneous zoom shooting of multiple moving objects (persons and bottles) in a natural outdoor scene. Moreover, our system demonstrates high robustness to target loss and crossing situations.

Polarization-based optical characterization for color texture analysis and segmentation

• Hand-crafted texture characterization based on light polarization signatures. • Image segmentation relying on pixel based circular polarization signature. • Multispectral image (185 bands) with five total variation planes. • The Karhunen-Loeve transform and then k-means were applied to the 185 bands. • Superior segmentation accuracy (compared with classic k-means on RGB). Texture characterization is very useful for automatic analysis of object surface images for a plethora of applications in medicine, agriculture, industry or remote sensing. Various texture characterization techniques exist, from the classical Haralick descriptors, Gabor filters, local binary patterns to automatically-extracted features using machine learning models. We propose a new hand-crafted texture characterization technique, based on light polarization property, by deploying a circular polarization filter (rotated from 0° to 360° in steps of 10°) in the image acquisition process. The hypothesis is that different materials and surfaces will exhibit different polarization signatures defined as pixel values variation as a function of polarization angle. Such polarization signature is able to locally characterize texture as a consequence of light reflections captured in every pixel due to the texture intrinsic variations. We show the usefulness of our approach for surface/material classification for the purpose of color image segmentation of natural outdoor scenes.

Digital image processing methods for estimating leaf area of cucumber plants

Increasingly <span>emerging technologies in agriculture such as computer vision, artificial intelligence technology, not only make it possible to increase production. To minimize the negative impact on climate and the environment but also to conserve resources. A key task of these technologies is to monitor the growth of plants online with a high accuracy rate and in non-destructive manners. It is known that leaf area (LA) is one of the most important growth indexes in plant growth monitoring system. Unfortunately, to estimate the LA in natural outdoor scenes (the presence of occlusion or overlap area) with a high accuracy rate is not easy and it still remains a big challenge in eco-physiological studies. In this paper, two accurate and non-destructive approaches for estimating the LA were proposed with top-view and side-view images, respectively. The proposed approaches successfully extract the skeleton of cucumber plants in red, green, and blue (RGB) images and estimate the LA of cucumber plants with high precision. The results were validated by comparing with manual measurements. The experimental results of our proposed algorithms achieve 97.64% accuracy in leaf segmentation, and the relative error in LA estimation varies from 3.76% to 13.00%, which could meet the requirements of plant growth monitoring </span>systems.

Semantic Segmentation of Coastal Zone on Airborne Lidar Bathymetry Point Clouds

Large-scale semantic segmentation point cloud is an ongoing research topic for on-land environments. However, there is a rare deep learning research study for the sub-surface environment. Although, PointNet and its successor PointNet++ have become the cornerstone of point cloud segmentation. However, these techniques handle a relatively small number of points. This poses a natural difficulty in a large spatial scene with millions of possible points. In particular, for shallow water of coastal zone, the small number of points where the seabed and water surface meet, close points may belong to different classes. In our work, we present the semantic segmentation on a large-scale airborne Lidar bathymetry (ALB) point cloud containing millions of sample points into two classes of water surface and seabed with the voxel sampling pre-processing (VSP) approach. The proposed approach will allow us to capture the complicated outdoor natural scene components of water surface and seabed more accurately and more realistic through nonuniform voxelization in the mixture of dense and sparse points of the ALB point cloud. The performance of validation results show improvement in a per-point accuracy of 72.45% compared with other state-of-the-art deep learning-based methods.

A Simultaneous Multi-Object Zooming System Using an Ultrafast Pan-Tilt Camera

This paper presents a novel dual-camera system that can simultaneously capture zoomed-in images using an ultrafast pan-tilt camera and a fixed wide-view camera using deep learning methods. An ultrafast pan-tilt camera can function as multiple virtual pan-tilt cameras by synchronizing a high-frame-rate zooming-view camera and an ultrafast pan-tilt mirror device that can switch over 500 different views in a second. A wide-view camera can obtain images in a fixed view in which multiple targets to be tracked are captured in low resolution and then recognized by processing the images using deep learning methods at a rate of dozens of frames per second. Based on the positions of all targets, recognized by the wide-view camera, the pan and tilt angles of multiple pan-tilt cameras are virtually controlled using an ultrafast pan-tilt camera through multithread viewpoint control to simultaneously capture the zoomed-in images of all targets. Our developed system can operate 10 virtual pan-tilt cameras (25 fps) with multithread viewpoint control and 4 ms time granularity in synchronization with convolutional neural-network-based recognition model operating at 25 fps, which is accelerated by a general-purpose computing on graphics processing units. The effectiveness of our system was demonstrated by the results of several experiments conducted on simultaneous zoom shooting of multiple running objects (persons and cars) in the range of approximately 80 m or higher in a natural outdoor scene, which was formerly too wide for a single fixed camera to capture clearly and simultaneously.

The Spatial Frequency Content of Urban and Indoor Environments as a Potential Risk Factor for Myopia Development.

PurposeTo examine the hypothesis that the spatial frequency spectra of urban and indoor environments differ from the natural environment in ways that may promote the development of myopia.MethodsA total of 814 images were analyzed from three datasets; University of California Berkeley (UCB), University of Texas (UT), and Botswana (UPenn). Images were processed in Matlab (Mathworks Inc) to map the camera color characteristics to human cone sensitivities. From the photopic luminance images generated, two-dimensional spatial frequency (SF) spectra were calculated and converted to one-dimensional spectra by rotational averaging. The spatial filtering profile of a 0.4 Bangerter foil, which has been shown to induce myopia experimentally, was also determined.ResultsThe SF slope for natural scenes followed the recognized 1/fα relationship with mean slopes of −1.08, −0.90, and −1.04 for the UCB, UT and UPenn image sets, respectively. Indoor scenes had a significantly steeper slope (−1.48, UCB; −1.52, UT; P < 0.0001). Urban environments showed an intermediate slope (−1.29, UCB; −1.22, UT) that was significantly different from the slopes derived from the natural scenes (P < 0.0001). The change in SF content between natural outdoor scenes and indoors was comparable to that induced by a 0.4 Bangerter foil, which reduced the SF slope of a natural scene from −0.88 to −1.47.ConclusionsCompared to natural outdoor images, man-made outdoor and indoor environments have spatial frequency characteristics similar to those known to induce form-deprivation myopia in animal models. The spatial properties of the man-made environment may be one of the missing drivers of the human myopia epidemic.

Understanding and Predicting the Memorability of Outdoor Natural Scenes

Memorability measures how easily an image is to be memorized after glancing, which may contribute to designing magazine covers, tourism publicity materials, and so forth. Recent works have shed light on the visual features that make generic images, object images or face photographs memorable. However, these methods are not able to effectively predict the memorability of outdoor natural scene images. To overcome this shortcoming of previous works, in this paper, we provide an attempt to answer: “what exactly makes outdoor natural scenes memorable”. To this end, we first establish a large-scale outdoor natural scene image memorability (LNSIM) database, containing 2,632 outdoor natural scene images with their ground truth memorability scores and the multi-label scene category annotations. Then, similar to previous works, we mine our database to investigate how low-, middle- and high-level handcrafted features affect the memorability of outdoor natural scenes. In particular, we find that the high-level feature of scene category is rather correlated with outdoor natural scene memorability, and the deep features learnt by deep neural network (DNN) are also effective in predicting the memorability scores. Moreover, combining the deep features with the category feature can further boost the performance of memorability prediction. Therefore, we propose an end-to-end DNN based outdoor natural scene memorability (DeepNSM) predictor, which takes advantage of the learned category-related features. Then, the experimental results validate the effectiveness of our DeepNSM model, exceeding the state-of-the-art methods. Finally, we try to understand the reason of the good performance for our DeepNSM model, and also study the cases that our DeepNSM model succeeds or fails to accurately predict the memorability of outdoor natural scenes.

Dehazing Evaluation: Real-World Benchmark Datasets, Criteria, and Baselines

On benchmark images, modern dehazing methods are able to achieve very comparable results whose differences are too subtle for people to qualitatively judge. Thus, it is imperative to adopt quantitative evaluation on a vast number of hazy images. However, existing quantitative evaluation schemes are not convincing due to a lack of appropriate datasets and poor correlations between metrics and human perceptions. In this work, we attempt to address these issues, and we make two contributions. First, we establish two benchmark datasets, i.e., the BEnchmark Dataset for Dehazing Evaluation (BeDDE) and the EXtension of the BeDDE (exBeDDE), which had been lacking for a long period of time. The BeDDE is used to evaluate dehazing methods via full reference image quality assessment (FR-IQA) metrics. It provides hazy images, clear references, haze level labels, and manually labeled masks that indicate the regions of interest (ROIs) in image pairs. The exBeDDE is used to assess the performance of dehazing evaluation metrics. It provides extra dehazed images and subjective scores from people. To the best of our knowledge, the BeDDE is the first dehazing dataset whose image pairs were collected in natural outdoor scenes without any simulation. Second, we provide a new insight that dehazing involves two separate aspects, i.e., visibility restoration and realness restoration, which should be evaluated independently; thus, to characterize them, we establish two criteria, i.e., the visibility index (VI) and the realness index (RI), respectively. The effectiveness of the criteria is verified through extensive experiments. Furthermore, 14 representative dehazing methods are evaluated as baselines using our criteria on BeDDE. Our datasets and relevant code are available at https://github.com/xiaofeng94/BeDDE-for-defogging.

End-to-End Single Image Fog Removal Using Enhanced Cycle Consistent Adversarial Networks

Single image defogging is a classical and challenging problem in computer vision. Existing methods towards this problem mainly include handcrafted priors based methods that rely on the use of the atmospheric degradation model and learning based approaches that require paired fog-fogfree training example images. In practice, however, prior-based methods are prone to failure due to their own limitations and paired training data are extremely difficult to acquire. Inspired by the principle of CycleGAN network, we have developed an end-to-end learning system that uses unpaired fog and fogfree training images, adversarial discriminators and cycle consistency losses to automatically construct a fog removal system. Similar to CycleGAN, our system has two transformation paths; one maps fog images to a fogfree image domain and the other maps fogfree images to a fog image domain. Instead of one stage mapping, our system uses a two stage mapping strategy in each transformation path to enhance the effectiveness of fog removal. Furthermore, we make explicit use of prior knowledge in the networks by embedding the atmospheric degradation principle and a sky prior for mapping fogfree images to the fog images domain. In addition, we also contribute the first real world nature fog-fogfree image dataset for defogging research. Our multiple real fog images dataset (MRFID) contains images of 200 natural outdoor scenes. For each scene, there are one clear image and corresponding four foggy images of different fog densities manually selected from a sequence of images taken by a fixed camera over the course of one year. Qualitative and quantitative comparison against several state-of-the-art methods on both synthetic and real world images demonstrate that our approach is effective and performs favorably for recovering a clear image from a foggy image.

Learned compact local feature descriptor for TLS-based geodetic monitoring of natural outdoor scenes

Abstract. The advantages of terrestrial laser scanning (TLS) for geodetic monitoring of man-made and natural objects are not yet fully exploited. Herein we address one of the open challenges by proposing feature-based methods for identification of corresponding points in point clouds of two or more epochs. We propose a learned compact feature descriptor tailored for point clouds of natural outdoor scenes obtained using TLS. We evaluate our method both on a benchmark data set and on a specially acquired outdoor dataset resembling a simplified monitoring scenario where we successfully estimate 3D displacement vectors of a rock that has been displaced between the scans. We show that the proposed descriptor has the capacity to generalize to unseen data and achieves state-of-the-art performance while being time efficient at the matching step due the low dimension.

LEARNED COMPACT LOCAL FEATURE DESCRIPTOR FOR TLS-BASED GEODETIC MONITORING OF NATURAL OUTDOOR SCENES

Abstract. The advantages of terrestrial laser scanning (TLS) for geodetic monitoring of man-made and natural objects are not yet fully exploited. Herein we address one of the open challenges by proposing feature-based methods for identification of corresponding points in point clouds of two or more epochs. We propose a learned compact feature descriptor tailored for point clouds of natural outdoor scenes obtained using TLS. We evaluate our method both on a benchmark data set and on a specially acquired outdoor dataset resembling a simplified monitoring scenario where we successfully estimate 3D displacement vectors of a rock that has been displaced between the scans. We show that the proposed descriptor has the capacity to generalize to unseen data and achieves state-of-the-art performance while being time efficient at the matching step due the low dimension.

Design and realization of relic augmented reality system of integration positioning and posture sensing technology

On the purpose of presenting the old appearance of the relics through digitalization, and overlapping the virtual scenes with the actual scenes at the relic site, in this paper, we introduced the positioning technology, posture sensing technology and system development technology, put forward constructing cultural relic tourism platform based on integration positioning and posture sensing technology, we conducted detailed research and analysis on the users’ experienced process of cultural tourism, designed a relic augmented reality system of integration positioning and posture sensing technology. This augmented reality system mainly utilizes positioning technology to guide the users to the correct location of the relic on the corresponding map, and then overlaps the virtual object with the real relic, achieveing a 360-degree view of the overlapping effect, and the presentation effect of near-small, far-big. The system mainly employs Unity to develop the system and realize the above system on mobile terminal. It is no longer limited to a fixed point experience environment, and is suitable for outdoor natural scenes, it makes a breakthrough on the traditional overlapping of virtual scenes and real scenes, realizes the precise overlap of virtual 3D scenes with actual images, and enables the users to feel the vicissitudes of history along the movement of the mobile device in outdoor natural scenes, so as to inherit the history and culture, enrich the information and add some fun to the displayed scene, it has the advantages of bringing people more immersive feelings compared with the traditional virtual display platform.

Software to convert terrestrial LiDAR scans of natural environments into photorealistic meshes

The introduction of 3D scanning has strongly influenced environmental sciences. If the resulting point clouds can be transformed into polygon meshes, a vast range of visualisation and analysis tools can be applied. But extracting accurate meshes from large point clouds gathered in natural environments is not trivial, requiring a suite of customisable processing steps. We present Habitat3D, an open source software tool to generate photorealistic meshes from registered point clouds of natural outdoor scenes. We demonstrate its capability by extracting meshes of different environments: 8,800 m2 grassland featuring several Eucalyptus trees (combining 9 scans and 41,989,885 data points); 1,018 m2 desert densely covered by vegetation (combining 56 scans and 192,223,621 data points); a well-structured garden; and a rough, volcanic surface. The resultant reconstructions accurately preserve all spatial features with millimetre accuracy whilst reducing the memory load by up to 98.5%. This enables rapid visualisation of the environments using off-the-shelf game engines and graphics hardware.

The Nature and Timing of Tele-Pseudoscopic Experiences.

Interchanging the left and right eye views of a scene (pseudoscopic viewing) has been reported to produce vivid stereoscopic effects under certain conditions. In two separate field studies, we examined the experiences of 124 observers (76 in Study 1 and 48 in Study 2) while pseudoscopically viewing a distant natural outdoor scene. We found large individual differences in both the nature and the timing of their pseudoscopic experiences. While some observers failed to notice anything unusual about the pseudoscopic scene, most experienced multiple pseudoscopic phenomena, including apparent scene depth reversals, apparent object shape reversals, apparent size and flatness changes, apparent reversals of border ownership, and even complex illusory foreground surfaces. When multiple effects were experienced, patterns of co-occurrence suggested possible causal relationships between apparent scene depth reversals and several other pseudoscopic phenomena. The latency for experiencing pseudoscopic phenomena was found to correlate significantly with observer visual acuity, but not stereoacuity, in both studies.

SET: a pupil detection method using sinusoidal approximation.

Mobile eye-tracking in external environments remains challenging, despite recent advances in eye-tracking software and hardware engineering. Many current methods fail to deal with the vast range of outdoor lighting conditions and the speed at which these can change. This confines experiments to artificial environments where conditions must be tightly controlled. Additionally, the emergence of low-cost eye tracking devices calls for the development of analysis tools that enable non-technical researchers to process the output of their images. We have developed a fast and accurate method (known as “SET”) that is suitable even for natural environments with uncontrolled, dynamic and even extreme lighting conditions. We compared the performance of SET with that of two open-source alternatives by processing two collections of eye images: images of natural outdoor scenes with extreme lighting variations (“Natural”); and images of less challenging indoor scenes (“CASIA-Iris-Thousand”). We show that SET excelled in outdoor conditions and was faster, without significant loss of accuracy, indoors. SET offers a low cost eye-tracking solution, delivering high performance even in challenging outdoor environments. It is offered through an open-source MATLAB toolkit as well as a dynamic-link library (“DLL”), which can be imported into many programming languages including C# and Visual Basic in Windows OS (www.eyegoeyetracker.co.uk).

Robust Sign Recognition System at Subway Stations Using Verification Knowledge

In this paper, we present a walking guidance system for the visually impaired for use at subway stations. This system, which is based on environmental knowledge, automatically detects and recognizes both exit numbers and arrow signs from natural outdoor scenes. The visually impaired can, therefore, utilize the system to find their own way (for example, using exit numbers and the directions provided) through a subway station. The proposed walking guidance system consists mainly of three stages: (a) sign detection using the MCT-based AdaBoost technique, (b) sign recognition using support vector machines and hidden Markov models, and (c) three verification techniques to discriminate between signs and non-signs. The experimental results indicate that our sign recognition system has a high performance with a detection rate of 98%, a recognition rate of 99.5%, and a false-positive error rate of 0.152.

Prediction of perceived fog density and defogging of natural foggy images

The perception of outdoor natural scenes is important for successfully participating in visual activities. When fog is present, however, visibility can be severely degraded. We have created a perceptual fog density index that seeks to predict the degree of visibility of a foggy scene using ‘fog aware’ features learned from a representative database of foggy and fog-free images. The features that define the fog density index derive from a spatial natural scene statistics (NSS) model and from observed characteristics of foggy images. We have found that the statistics of fog aware features consistently change as a function of fog density. A perceptual fog density model was derived by collecting patches from natural foggy and fog-free images, then computing pertinent NSS fog aware features from these patches to develop a perceptual fog density prediction model. A multivariate Gaussian (MVG) distribution is used to form a probabilistic feature model. In practice the perceived fog density of an arbitrary test image is predicted using a Mahalanobis-like distance measure between the ‘fog aware’ statistics of the test image and the MVG models obtained from the natural foggy and fog-free images. We also conducted a human study of perceived fog density. When applied to 100 natural foggy images, the predicted perceived fog density was found to correlate well with the human judgments of fog density. We have also found that the fog aware statistical features can be used to defog and thereby enhance the visibility of foggy scenes. We first produced white-balanced and contrast-enhanced images from a foggy image using the predicted visibility degree. We then selectively filtered these images with fog aware weighted maps representing distances from the statistics of fog-free images at each fog aware feature. Finally, we applied a Laplacian multiscale pyramidal refinement to achieve a halo-free defogged image.

Multiscale analysis of depth images from natural scenes: Scaling in the depth of the woods

We analyze an ensemble of images from outdoor natural scenes and consisting of pairs of a standard gray-level luminance image associated with a depth image of the same scene, delivered by a recently introduced low-cost sensor for joint imaging of depth and luminance. We specially focus on statistical analysis of multiscale and fractal properties in the natural images. Two methodologies are implemented for this purpose, and examining the distribution of contrast upon coarse-graining at increasing scales, and the orientationally averaged power spectrum tied to spatial frequencies. Both methodologies confirm, on another independent dataset here, the presence of fractal scale invariance in the luminance natural images, as previously reported. Both methodologies here also reveal the presence of fractal scale invariance in the novel data formed by depth images from natural scenes. The multiscale analysis is confronted on luminance images and on the novel depth images together with an analysis of their statistical correlation. The results, especially the new results on the multiscale analysis of depth images, consolidate the importance and extend the multiplicity of aspects of self-similarity and fractal scale invariance properties observable in the constitution of images from natural scenes. Such results are useful to better understanding and modeling of the (multiscale) structure of images from natural scenes, with relevance to image processing algorithms and to visual perception. The approach also contains potentialities for the fractal characterization of three-dimensional natural structures and their interaction with light.

Outdoor Dynamic 3-D Scene Reconstruction

Existing systems for 3-D reconstruction from multiple view video use controlled indoor environments with uniform illumination and backgrounds to allow accurate segmentation of dynamic foreground objects. In this paper, we present a portable system for 3-D reconstruction of dynamic outdoor scenes that require relatively large capture volumes with complex backgrounds and nonuniform illumination. This is motivated by the demand for 3-D reconstruction of natural outdoor scenes to support film and broadcast production. Limitations of existing multiple view 3-D reconstruction techniques for use in outdoor scenes are identified. Outdoor 3-D scene reconstruction is performed in three stages: 1) 3-D background scene modeling using spherical stereo image capture; 2) multiple view segmentation of dynamic foreground objects by simultaneous video matting across multiple views; and 3) robust 3-D foreground reconstruction and multiple view segmentation refinement in the presence of segmentation and calibration errors. Evaluation is performed on several outdoor productions with complex dynamic scenes including people and animals. Results demonstrate that the proposed approach overcomes limitations of previous indoor multiple view reconstruction approaches enabling high-quality free-viewpoint rendering and 3-D reference models for production.