Complex Urban Scenes Research Articles

Building Façade Recognition Using Oblique Aerial Images

This study proposes a method to recognize façades from large-scale urban scenes based on multi-level image features utilizing a recently developed oblique aerial photogrammetry technique. The method involves the use of multi-level image features, a bottom-up feature extraction procedure to produce regions of interest through monoscopic analysis, and then a coarse-to-fine feature matching strategy to characterise and match the regions in a stereoscopic model. Feature extraction from typical urban Manhattan scenes is based on line segments. Windows are re-organised based on the spatial constraints of line segments and the homogeneous structure of the spectrum. Façades as regions of interest are successfully constructed with a remarkable single edge and evidence from windows to get rid of occlusion. Feature matching is hierarchically performed beginning from distinctive facades and regularly distributed windows to the sub-pixel point primitives. The proposed strategy can effectively solve ambiguity and multi-solution problems in the complex urban scene matching process, particularly repetitive and poor-texture façades in oblique view.

Particle filtering methods for georeferencing panoramic image sequence in complex urban scenes

Georeferencing image sequences is critical for mobile mapping systems. Traditional methods such as bundle adjustment need adequate and well-distributed ground control points (GCP) when accurate GPS data are not available in complex urban scenes. For applications of large areas, automatic extraction of GCPs by matching vehicle-born image sequences with geo-referenced ortho-images will be a better choice than intensive GCP collection with field surveying. However, such image matching generated GCP’s are highly noisy, especially in complex urban street environments due to shadows, occlusions and moving objects in the ortho images. This study presents a probabilistic solution that integrates matching and localization under one framework. First, a probabilistic and global localization model is formulated based on the Bayes’ rules and Markov chain. Unlike many conventional methods, our model can accommodate non-Gaussian observation. In the next step, a particle filtering method is applied to determine this model under highly noisy GCP’s. Owing to the multiple hypotheses tracking represented by diverse particles, the method can balance the strength of geometric and radiometric constraints, i.e., drifted motion models and noisy GCP’s, and guarantee an approximately optimal trajectory. Carried out tests are with thousands of mobile panoramic images and aerial ortho-images. Comparing with the conventional extended Kalman filtering and a global registration method, the proposed approach can succeed even under more than 80% gross errors in GCP’s and reach a good accuracy equivalent to the traditional bundle adjustment with dense and precise control.

Road Centerline Extraction in Complex Urban Scenes From LiDAR Data Based on Multiple Features

Automatic extraction of roads from images of complex urban areas is a very difficult task due to the occlusions and shadows of contextual objects, and complicated road structures. As light detection and ranging (LiDAR) data explicitly contain direct 3-D information of the urban scene and are less affected by occlusions and shadows, they are a good data source for road detection. This paper proposes to use multiple features to detect road centerlines from the remaining ground points after filtering. The main idea of our method is to effectively detect smooth geometric primitives of potential road centerlines and to separate the connected nonroad features (parking lots and bare grounds) from the roads. The method consists of three major steps, i.e., spatial clustering based on multiple features using an adaptive mean shift to detect the center points of roads, stick tensor voting to enhance the salient linear features, and a weighted Hough transform to extract the arc primitives of the road centerlines. In short, we denote our method as Mean shift, Tensor voting, Hough transform (MTH). We evaluated the method using the Vaihingen and Toronto data sets from the International Society for Photogrammetry and Remote Sensing Test Project on Urban Classification and 3-D Building Reconstruction. The completeness of the extracted road network on the Vaihingen data and the Toronto data are 81.7% and 72.3%, respectively, and the correctness are 88.4% and 89.2%, respectively, yielding the best performance compared with template matching and phase-coded disk methods.

Automatic Building Extraction From LIDAR Data Covering Complex Urban Scenes

Abstract. This paper presents a new method for segmentation of LIDAR point cloud data for automatic building extraction. Using the ground height from a DEM (Digital Elevation Model), the non-ground points (mainly buildings and trees) are separated from the ground points. Points on walls are removed from the set of non-ground points by applying the following two approaches: If a plane fitted at a point and its neighbourhood is perpendicular to a fictitious horizontal plane, then this point is designated as a wall point. When LIDAR points are projected on a dense grid, points within a narrow area close to an imaginary vertical line on the wall should fall into the same grid cell. If three or more points fall into the same cell, then the intermediate points are removed as wall points. The remaining non-ground points are then divided into clusters based on height and local neighbourhood. One or more clusters are initialised based on the maximum height of the points and then each cluster is extended by applying height and neighbourhood constraints. Planar roof segments are extracted from each cluster of points following a region-growing technique. Planes are initialised using coplanar points as seed points and then grown using plane compatibility tests. If the estimated height of a point is similar to its LIDAR generated height, or if its normal distance to a plane is within a predefined limit, then the point is added to the plane. Once all the planar segments are extracted, the common points between the neghbouring planes are assigned to the appropriate planes based on the plane intersection line, locality and the angle between the normal at a common point and the corresponding plane. A rule-based procedure is applied to remove tree planes which are small in size and randomly oriented. The neighbouring planes are then merged to obtain individual building boundaries, which are regularised based on long line segments. Experimental results on ISPRS benchmark data sets show that the proposed method offers higher building detection and roof plane extraction rates than many existing methods, especially in complex urban scenes.

An energy minimization approach to automated extraction of regular building footprints from airborne LiDAR data

Abstract. This paper presents an automated approach to the extraction of building footprints from airborne LiDAR data based on energy minimization. Automated 3D building reconstruction in complex urban scenes has been a long-standing challenge in photogrammetry and computer vision. Building footprints constitute a fundamental component of a 3D building model and they are useful for a variety of applications. Airborne LiDAR provides large-scale elevation representation of urban scene and as such is an important data source for object reconstruction in spatial information systems. However, LiDAR points on building edges often exhibit a jagged pattern, partially due to either occlusion from neighbouring objects, such as overhanging trees, or to the nature of the data itself, including unavoidable noise and irregular point distributions. The explicit 3D reconstruction may thus result in irregular or incomplete building polygons. In the presented work, a vertex-driven Douglas-Peucker method is developed to generate polygonal hypotheses from points forming initial building outlines. The energy function is adopted to examine and evaluate each hypothesis and the optimal polygon is determined through energy minimization. The energy minimization also plays a key role in bridging gaps, where the building outlines are ambiguous due to insufficient LiDAR points. In formulating the energy function, hard constraints such as parallelism and perpendicularity of building edges are imposed, and local and global adjustments are applied. The developed approach has been extensively tested and evaluated on datasets with varying point cloud density over different terrain types. Results are presented and analysed. The successful reconstruction of building footprints, of varying structural complexity, along with a quantitative assessment employing accurate reference data, demonstrate the practical potential of the proposed approach.

3D change detection at street level using mobile laser scanning point clouds and terrestrial images

Automatic change detection and geo-database updating in the urban environment are difficult tasks. There has been much research on detecting changes with satellite and aerial images, but studies have rarely been performed at the street level, which is complex in its 3D geometry. Contemporary geo-databases include 3D street-level objects, which demand frequent data updating. Terrestrial images provides rich texture information for change detection, but the change detection with terrestrial images from different epochs sometimes faces problems with illumination changes, perspective distortions and unreliable 3D geometry caused by the lack of performance of automatic image matchers, while mobile laser scanning (MLS) data acquired from different epochs provides accurate 3D geometry for change detection, but is very expensive for periodical acquisition. This paper proposes a new method for change detection at street level by using combination of MLS point clouds and terrestrial images: the accurate but expensive MLS data acquired from an early epoch serves as the reference, and terrestrial images or photogrammetric images captured from an image-based mobile mapping system (MMS) at a later epoch are used to detect the geometrical changes between different epochs. The method will automatically mark the possible changes in each view, which provides a cost-efficient method for frequent data updating. The methodology is divided into several steps. In the first step, the point clouds are recorded by the MLS system and processed, with data cleaned and classified by semi-automatic means. In the second step, terrestrial images or mobile mapping images at a later epoch are taken and registered to the point cloud, and then point clouds are projected on each image by a weighted window based z-buffering method for view dependent 2D triangulation. In the next step, stereo pairs of the terrestrial images are rectified and re-projected between each other to check the geometrical consistency between point clouds and stereo images. Finally, an over-segmentation based graph cut optimization is carried out, taking into account the color, depth and class information to compute the changed area in the image space. The proposed method is invariant to light changes, robust to small co-registration errors between images and point clouds, and can be applied straightforwardly to 3D polyhedral models. This method can be used for 3D street data updating, city infrastructure management and damage monitoring in complex urban scenes.

Contextual classification of lidar data and building object detection in urban areas

In this work we address the task of the contextual classification of an airborne LiDAR point cloud. For that purpose, we integrate a Random Forest classifier into a Conditional Random Field (CRF) framework. It is a flexible approach for obtaining a reliable classification result even in complex urban scenes. In this way, we benefit from the consideration of context on the one hand and from the opportunity to use a large amount of features on the other hand. Considering the interactions in our experiments increases the overall accuracy by 2%, though a larger improvement becomes apparent in the completeness and correctness of some of the seven classes discerned in our experiments. We compare the Random Forest approach to linear models for the computation of unary and pairwise potentials of the CRF, and investigate the relevance of different features for the LiDAR points as well as for the interaction of neighbouring points. In a second step, building objects are detected based on the classified point cloud. For that purpose, the CRF probabilities for the classes are plugged into a Markov Random Field as unary potentials, in which the pairwise potentials are based on a Potts model. The 2D binary building object masks are extracted and evaluated by the benchmark ISPRS Test Project on Urban Classification and 3D Building Reconstruction. The evaluation shows that the main buildings (larger than 50m2) can be detected very reliably with a correctness larger than 96% and a completeness of 100%.

Summarizing high-level scene behavior

We present several novel techniques to summarize the high-level behavior in surveillance video. Our proposed methods can employ either optical flow or trajectories as input, and incorporate spatial and temporal information together, which improve upon existing approaches for summarization. To begin, we extract common pathway regions by performing graph-based clustering on similarity matrices describing the relationships between location/orientation states. We then employ the activities along the pathway regions to extract the aggregate behavioral patterns throughout scenes. We show how our summarization methods can be applied to detect anomalies, retrieve video clips of interest, and generate adaptive-speed summary videos. We examine our approaches on multiple complex urban scenes and present experimental results.

Influence of front light configuration on the visual conspicuity of motorcycles

A recent study (Cavallo and Pinto, 2012) showed that daytime running lights (DRLs) on cars create “visual noise” that interferes with the lighting of motorcycles and affects their visual conspicuity. In the present experiment, we tested three conspicuity enhancements designed to improve motorcycle detectability in a car-DRL environment: a triangle configuration (a central headlight plus two lights located on the rearview mirrors), a helmet configuration (a light located on the motorcyclist's helmet in addition to the central headlight), and a single central yellow headlight. These three front-light configurations were evaluated in comparison to the standard configuration (a single central white headlight). Photographs representing complex urban traffic scenes were presented briefly (for 250ms). The results revealed better motorcycle-detection performance for both the yellow headlight and the helmet configuration than for the standard configuration. The findings suggest some avenues for defining a new visual signature for motorcycles in car-DRL environments.

CONDITIONAL RANDOM FIELDS FOR LIDAR POINT CLOUD CLASSIFICATION IN COMPLEX URBAN AREAS

Abstract. In this paper, we investigate the potential of a Conditional Random Field (CRF) approach for the classification of an airborne LiDAR (Light Detection And Ranging) point cloud. This method enables the incorporation of contextual information and learning of specific relations of object classes within a training step. Thus, it is a powerful approach for obtaining reliable results even in complex urban scenes. Geometrical features as well as an intensity value are used to distinguish the five object classes building, low vegetation, tree, natural ground, and asphalt ground. The performance of our method is evaluated on the dataset of Vaihingen, Germany, in the context of the 'ISPRS Test Project on Urban Classification and 3D Building Reconstruction'. Therefore, the results of the 3D classification were submitted as a 2D binary label image for a subset of two classes, namely building and tree.

TREES DETECTION FROM LASER POINT CLOUDS ACQUIRED IN DENSE URBAN AREAS BY A MOBILE MAPPING SYSTEM

Abstract. 3D reconstruction of trees is of great interest in large-scale 3D city modelling. Laser scanners provide geometrically accurate 3D point clouds that are very useful for object recognition in complex urban scenes. Trees often cause important occlusions on building façades. Their recognition can lead to occlusion maps that are useful for many façade oriented applications such as visual based localisation and automatic image tagging. This paper proposes a pipeline to detect trees in point clouds acquired in dense urban areas with only laser informations (x,y, z coordinates and intensity). It is based on local geometric descriptors computed on each laser point using a determined neighbourhood. These descriptors describe the local shape of objects around every 3D laser point. A projection of these values on a 2D horizontal accumulation space followed by a combination of morphological filters provides individual tree clusters. The pipeline is evaluated and the results are presented on a set of one million laser points using a man made ground truth.

Event classification for vehicle navigation system by regional optical flow analysis

We address the problem of event classification for intelligent vehicle navigation system from video sequences acquired by a front mounted camera in complex urban scenes. Although in normal driving condition, large variety of events could be found and be preferably attached to an alerting system in a vehicle, there have been relatively narrow research activities on driving scene analysis, for example, finding local information such as lanes, pedestrians, traffic signs or light detections. Yet, the above-mentioned methods only provide limited performance due to many challenges in normal urban driving conditions, i.e. complex background, inhomogeneous illumination, occlusion, etc. In this paper, we tackle the problem of classification of various events by learning regional optical flows to detect some important events (very frequent occurring and involving riskiness on driving) using low cost front mounted camera equipment. We approached the problem as follows: First, we present an optical flow-based event detection method by regional significance analysis with the introduction of a novel significance map based on regional histograms of flow vectors; Second, we present a global and a local method to robustly detect ego-motion-based events and target-motion-based events. In our experiments, we achieved classification accuracy about 91% on average tested with two classifiers (Bayesian and SVM). We also show the performance of the method in terms of computational complexity achieving about 14.3 fps on a laptop computer with Intel Pentium 1.2 Ghz.

Are car daytime running lights detrimental to motorcycle conspicuity?

For a long time, motorcycles were the only vehicles with daytime running lights (DRLs), but this conspicuity advantage has been questioned due to the rapidly increasing introduction of DRLs on cars as well. The present experiment was designed to assess effects of car DRLs on motorcycle perception in a situation that specifically brought attentional conspicuity to bear. Photographs representing complex urban traffic scenes were displayed briefly (250ms) to 24 participants who had to detect vulnerable road users (motorcyclists, cyclists, pedestrians) appearing at different locations and distances. Car DRLs hampered motorcycle perception compared to conditions where car lights were not on, especially when the motorcycle was at a greater distance from the observer and when it was located in the central part of the visual scene. Car DRLs also hampered the perception of cyclists and pedestrians. Although the globally positive safety effect of car DRLs is generally acknowledged, our study suggests that more attention should be paid to motorcyclists and other vulnerable road users when introducing car DRLs. Several means of improving motorcycle conspicuity in car DRL environments are discussed.

An Enhanced Background Estimation Algorithm for Vehicle Detection in Urban Traffic Scenes

This paper proposes a new background subtraction algorithm based on the sigma-delta filter, which is intended to be used in urban traffic scenes. The original sigma-delta algorithm is a very interesting alternative due to its high computational efficiency. However, the background model quickly degrades in complex urban scenes because it is easily “contaminated” by slow-moving or temporarily stopped vehicles. Then, subsequent foreground validation steps are needed to refine the foreground detection mask. Instead of requiring any subsequent processing steps or resorting to algorithms with higher computational cost, the proposed algorithm tries to achieve a more stable background model by introducing a confidence measurement for each pixel. This confidence measurement assists in a selective background-model updating mechanism at the pixel level. Experimental comparative tests and a quantitative performance evaluation over typical urban traffic sequences corroborate the benefits of the proposed algorithm.

Extraction and Geometrical Accuracy of Double-bounce Lines in High Resolution SAR Images

In TerraSAR-X spotlight data features of man-made objects in urban areas become visible. In a case study, this paper presents results of a precision and geometrical accuracy investigation of extracted double-bounce lines in SAR images. Two TerraSAR-X (TSX) acquisitions of a complex urban scene are compared to simulated SAR images. Using a new line detection method, double-bounce lines are extracted from simulated and real imagery with sub-pixel accuracy. The deviation of an extracted double-bounce line polygon with regard to both an interpolated line and ground truth is analyzed and discussed. First, it is shown that Gaussian functions model the deviation of the interpolated line from the polygon sufficiently well. Second, the absolute geometrical accuracy of the extracted double-bounce lines is evaluated with respect to ground truth. Results of the case study confirm that double-bounce lines provide reliable geometrical information, for instance, for building extraction and SAR image classification.

Texture analysis of IKONOS satellite imagery for urban land use and land cover classification

Traditional spectral-based methods of extracting urban land cover and land use information from remote sensing imagery have proven to be unsuitable for high spatial resolution images. Texture has been widely investigated as a supplement to spectral data for the analysis of complex urban scenes. This research evaluates the grey level co-occurrence matrix (GLCM) texture analysis technique and the maximum likelihood classification approach for the extraction of texture features to be combined with spectral data, as a method for obtaining more accurate urban land cover and land use information from high spatial resolution images. Classifications were performed on IKONOS imagery using three datasets: a spatial dataset consisting of three texture images (mean, homogeneity and dissimilarity), a spectral dataset consisting of four spectral images (red, green, blue and NIR) and a combination dataset (spatial and spectral). Results show that the combination dataset produced the highest overall classification accuracy of 86.1%, an improvement of 7.2% over the spectral dataset.

Early evoked gamma-band responses and phase synchronization elicited by target categorical visual object discrimination

Event Abstract Back to Event Early evoked gamma-band responses and phase synchronization elicited by target categorical visual object discrimination Nikolaos Tsarouchas1, 2, 3*, S. Kostopoulos1, A. Bezeriano1, G. Benedek2 and K. Benedek3 1 Department of Medical Physics, University of Patras, Greece 2 Department of Physiology, University of Szeged, Hungary 3 Department of Neurology, University of Szeged, Hungary Objective: Aim of this study is to investigate whether target categorical visual object discrimination processes modulate cortical gamma-band responses (GBR). Methods: We analyze the amplitude of evoked vs. induced and the phase-locking factor of γ-band responses (≥30Hz) as contrasted to event-related potentials (<30Hz), by means of the continuous wavelet transform, in an ultra-rapid visuocognitive categorical discrimination task. Visual stimuli comprised 500 briefly presented images of complex natural and urban scenes. Subjects had to categorize them as targets (containing different animal objects) by eliciting a motor response vs. equiprobable nontargets (containing only nonanimal objects) by suppressing it. Results: Statistical parametric time-frequency map analysis across subjects and channels identified a characteristic (80-180)ms×(30-50)Hz poststimulus region-of-interest over occipital visual cortex (channels O1/O2), within which significantly higher evoked γ-band amplitudes and phase-locking factors were established for target vs. nontarget categorical stimuli. Conclusions: Enhanced early evoked GBR most likely emerge from the interaction of bottom-up feedforward (visual-feature extracting and binding) with top-down attentional feedback (visual-feature weighting and integrating) processes due to phase synchronization across single-trials in the “resultant” activity of the underlying occipital cortical domains. Significance: Our study provides evidence that early occipital cortical GBR underlie already the representation of target visual objects at the categorical level and the discrimination of different categorical exemplars across higher-order classes of visual objects. Acknowledgments: This work was supported by a PhD Scholarship granted to the author by Greek State Scholarship Foundation (IKY) and a research grant of the University of Patras, Karatheodoris (2004-B411): Functional Laser Speckle Imaging (fLSI). Conference: 10th International Conference on Cognitive Neuroscience, Bodrum, Turkey, 1 Sep - 5 Sep, 2008. Presentation Type: Oral Presentation Topic: Brain Electrical Oscillations in Cognition Citation: Tsarouchas N, Kostopoulos S, Bezeriano A, Benedek G and Benedek K (2008). Early evoked gamma-band responses and phase synchronization elicited by target categorical visual object discrimination. Conference Abstract: 10th International Conference on Cognitive Neuroscience. doi: 10.3389/conf.neuro.09.2009.01.113 Copyright: The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers. They are made available through the Frontiers publishing platform as a service to conference organizers and presenters. The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated. Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed. For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions. Received: 03 Dec 2008; Published Online: 03 Dec 2008. * Correspondence: Nikolaos Tsarouchas, Department of Medical Physics, University of Patras, Patras, Greece, nick_tsarouchas@yahoo.com Login Required This action requires you to be registered with Frontiers and logged in. To register or login click here. Abstract Info Abstract The Authors in Frontiers Nikolaos Tsarouchas S. Kostopoulos A. Bezeriano G. Benedek K. Benedek Google Nikolaos Tsarouchas S. Kostopoulos A. Bezeriano G. Benedek K. Benedek Google Scholar Nikolaos Tsarouchas S. Kostopoulos A. Bezeriano G. Benedek K. Benedek PubMed Nikolaos Tsarouchas S. Kostopoulos A. Bezeriano G. Benedek K. Benedek Related Article in Frontiers Google Scholar PubMed Abstract Close Back to top Javascript is disabled. Please enable Javascript in your browser settings in order to see all the content on this page.

Real-Time Dense Stereo for Intelligent Vehicles

Stereo vision is an attractive passive sensing technique for obtaining three-dimensional (3-D) measurements. Recent hardware advances have given rise to a new class of real-time dense disparity estimation algorithms. This paper examines their suitability for intelligent vehicle (IV) applications. In order to gain a better understanding of the performance and the computational-cost tradeoff, the authors created a framework of real-time implementations. This consists of different methodical components based on single instruction multiple data (SIMD) techniques. Furthermore, the resulting algorithmic variations are compared with other publicly available algorithms. The authors argue that existing publicly available stereo data sets are not very suitable for the IV domain. Therefore, the authors' evaluation of stereo algorithms is based on novel realistically looking simulated data as well as real data from complex urban traffic scenes. In order to facilitate future benchmarks, all data used in this paper is made publicly available. The results from this study reveal that there is a considerable influence of scene conditions on the performance of all tested algorithms. Approaches that aim for (global) search optimization are more affected by this than other approaches. The best overall performance is achieved by the proposed multiple-window algorithm, which uses local matching and a left-right check for a robust error rejection. Timing results show that the simplest of the proposed SIMD variants are more than twice as fast than the most complex one. Nevertheless, the latter still achieves real-time processing speeds, while their average accuracy is at least equal to that of publicly available non-SIMD algorithms

A Global Infrared Image Synthesis Model for Large-Scale Complex Urban Scene

Most of infrared scene models deal with backgrounds, they can't simulate infrared urban scene. This paper presents a global infrared image synthesis model for large-scale complex urban scene which can be applied to various weather conditions and suitable for various parts of the urban scene. We first analysis the influence of all kinds of meteorological and environmental factors on the infrared characteristic of urban scene. Then based on principle of energy equilibrium we propose a global infrared image synthesis model for urban scene, discuss the specific forms for various parts of the scene and compute their surface temperature. As the rendering principle for infrared scene differs from that for visible scene, we propose a novel rendering criterion for infrared scene. Finally by the method of Gouraud Shading of Computer Graphics, infrared images of urban scene from different viewpoints at different time are presented. Field measurement further shows that our model is robust and feasible.

3-D Reconstruction of Urban Scenes from Aerial Stereo Imagery: A Focusing Strategy

A contribution to the automatic 3-D reconstruction of complex urban scenes from aerial stereo pairs is proposed. It consists of segmenting the scene into two different kinds of components: the ground and the above-ground objects. The above-ground objects are classified either as buildings or as vegetation. The idea is to define appropriate regions of interest in order to achieve a relevant 3-D reconstruction. For that purpose, a digital elevation model of the scene is first computed and segmented into above-ground regions using a Markov random field model. Then a radiometric analysis is used to classify above-ground regions as building or vegetation, leading to the determination of the final above-ground objects. The originality of the method is its ability to cope with extended above-ground areas, even in case of a sloping ground surface. This characteristic is necessary in a urban environment. Results are very robust to image and scene variability, and they enable the utilization of appropriate local 3-D reconstruction algorithms.