Tencent Street View Research Articles

Deep-learning generation of POI data with scene images

Point of interest (POI) is essential to urban scene understanding and location-based services. However, most of the POI data sets are collected manually on the spot, which is time-consuming and laborious. In this study, we propose a deep learning-based three-stage framework to automatically generate POI data sets from scene images by integrating instance segmentation, scene text recognition (STR), and multimodal technology. Firstly, we utilize an instance segmentation model to extract the region of interest (ROI) that contains POI text information from the scene images. Secondly, a STR method is used to locate and identify the text lines from the ROI. Thirdly, we develop a novel visual-linguistic multi-task classification model (VLMC) to classify ROIs and text lines through fusing text and image information. It is the first deep learning-based framework that allows generating POI information with different attributes (such as title, address, and tag) from the text lines of scene images and updating with high-performance models in the three-stage technique. In the experiments, we employ multiple STR data sets and annotated street view images for model training. The result shows that the deep learning-based framework can generate POI records from scene images with high accuracy (F1-score = 52.62%). Moreover, we find that the multi-modal VLMC model integrating the linguistic and visual embeddings has a higher accuracy in POI-generation than single-modal methods. We further use a trained framework to generate POI from Baidu Street View (BSV) images and Tencent Street View (TSV) images in Shenzhen, China, and ultimately obtain a long-term POI data set during 2013 – 2020 with 2,699,895 street view images. Of 815,616 records in the generated POI data set in 2020, 70.94% are covered by the existing Baidu POI data set of Shenzhen in 2013. This confirms the validity of the newly generated POI data set. These results demonstrate that the proposed deep-learning POI-generation framework and dataset can provide new insights for geographic data updating and urban scene understanding for fast growing cities. To facilitate future research, an implementation is made available at https://github.com/KampauCheung/scene-image-poi-generation.

Predicting the effect of street environment on residents' mood states in large urban areas using machine learning and street view images

BackgroundResearchers have demonstrated that the built environment is associated with mental health outcomes. However, evidence concerning the effects of street environments on mood in fast-growing Asian cities is scarce. Traditional questionnaires and interview methods are labor intensive and time consuming and pose challenges for accurately and efficiently evaluating the impact of urban-scale street environments on mood. ObjectiveThis study aims to use street view images and machine learning methods to model the impact of street environments on mood states in a large urban area in Guangzhou, China, and to assess the effect of different street view elements on mood. MethodsA total of 199,754 street view images of Guangzhou were captured from Tencent Street View, and street elements were extracted by pyramid scene parsing network. Data on six mood state indicators (motivated, happy, positive-social emotion, focused, relaxed, and depressed) were collected from 1590 participants via an online platform called Assessing the Effects of Street Views on Mood. A machine learning approach was proposed to predict the effects of street environment on mood in large urban areas in Guangzhou. A series of statistical analyses including stepwise regression, ridge regression, and lasso regression were conducted to assess the effects of street view elements on mood. ResultsStreets in urban fringe areas were more likely to produce motivated, happy, relaxed, and focused feelings in residents than those in city center areas. Conversely, areas in the city center, a high-density built environment, were more likely to produce depressive feelings. Street view elements have different effects on the six mood states. “Road” is a robust indicator positively correlated with the “motivated” indicator and negatively correlated with the “depressed” indicator. “Sky” is negatively associated with “positive-social emotion” and “depressed” but positively associated with “motivated”. “Building” is a negative predictor for the “focused” and “happy” indicator but is positively related to the “depressed” indicator, while “vegetation” and “terrain” are the variables most robustly and positively correlated with all positive moods. ConclusionOur findings can help urban designers identify crucial areas of the city for optimization, and they have practical implications for urban planners seeking to build urban environments that foster better mental health.

Big Data-Driven Pedestrian Analytics: Unsupervised Clustering and Relational Query Based on Tencent Street View Photographs

Recent technological advancements in geomatics and mobile sensing have led to various urban big data, such as Tencent street view (TSV) photographs; yet, the urban objects in the big dataset have hitherto been inadequately exploited. This paper aims to propose a pedestrian analytics approach named vectors of uncountable and countable objects for clustering and analysis (VUCCA) for processing 530,000 TSV photographs of Hong Kong Island. First, VUCCA transductively adopts two pre-trained deep models to TSV photographs for extracting pedestrians and surrounding pixels into generalizable semantic vectors of features, including uncountable objects such as vegetation, sky, paved pedestrian path, and guardrail and countable objects such as cars, trucks, pedestrians, city animals, and traffic lights. Then, the extracted pedestrians are semantically clustered using the vectors, e.g., for understanding where they usually stand. Third, pedestrians are semantically indexed using relations and activities (e.g., walking behind a guardrail, road-crossing, carrying a backpack, or walking a pet) for queries of unstructured photographic instances or natural language clauses. The experiment results showed that the pedestrians detected in the TSV photographs were successfully clustered into meaningful groups and indexed by the semantic vectors. The presented VUCCA can enrich eye-level urban features into computational semantic vectors for pedestrians to enable smart city research in urban geography, urban planning, real estate, transportation, conservation, and other disciplines.

Panoramic Street-Level Imagery in Data-Driven Urban Research: A Comprehensive Global Review of Applications, Techniques, and Practical Considerations

The release of Google Street View in 2007 inspired several new panoramic street-level imagery platforms including Apple Look Around, Bing StreetSide, Baidu Total View, Tencent Street View, Naver Street View, and Yandex Panorama. The ever-increasing global capture of cities in 360° provides considerable new opportunities for data-driven urban research. This paper provides the first comprehensive, state-of-the-art review on the use of street-level imagery for urban analysis in five research areas: built environment and land use; health and wellbeing; natural environment; urban modelling and demographic surveillance; and area quality and reputation. Panoramic street-level imagery provides advantages in comparison to remotely sensed imagery and conventional urban data sources, whether manual, automated, or machine learning data extraction techniques are applied. Key advantages include low-cost, rapid, high-resolution, and wide-scale data capture, enhanced safety through remote presence, and a unique pedestrian/vehicle point of view for analyzing cities at the scale and perspective in which they are experienced. However, several limitations are evident, including limited ability to capture attribute information, unreliability for temporal analyses, limited use for depth and distance analyses, and the role of corporations as image-data gatekeepers. Findings provide detailed insight for those interested in using panoramic street-level imagery for urban research.

Associations between trees and grass presence with childhood asthma prevalence using deep learning image segmentation and a novel green view index

Limitations of Normalized Difference Vegetation Index (NDVI) potentially contributed to the inconsistent findings of greenspace exposure and childhood asthma. The aim of this study was to use a novel greenness exposure assessment method, capable of overcoming the limitation of NDVI to determine the extent to which it was associated with asthma prevalence in Chinese children. During 2009–2013, a cross-sectional study of 59,754 children aged 2–17 years was conducted in northeast China. Tencent street view images surrounding participants’ schools were segmented by a deep learning model, and streetscape greenness was extracted. The green view index (GVI) was used to assign exposure and higher value indicates more green coverage. Mixed-effects logistic regression models were used to calculate the adjusted odds of asthma per interquartile range (IQR) increase of GVI for trees and grass. Participants were further stratified to investigate whether particulate matter with an aerodynamic diameter <2.5 μm (PM2.5) was a modifier. An IQR increase in GVI800m for trees was associated with lower adjusted odds of doctor-diagnosed asthma (OR: 0.76; 95%CI: 0.72–0.80) and current asthma (OR: 0.82; 95%CI: 0.75–0.89). An IQR increase in GVI800m for grass was associated with higher adjusted odds of doctor-diagnosed asthma (OR: 1.04; 95%CI: 1.00–1.08) and current asthma (OR: 1.08; 95%CI: 1.02–1.14). After stratification by PM2.5 exposure level, the negative association between trees and asthma, and the positive association between grass and asthma were observed only in low PM2.5 exposure levels (≤median: 56.23 μg/m3). Our results suggest that types of vegetation may play a role in the association between greenness exposure and childhood asthma. Exposure to trees may reduce the odds of childhood asthma, whereas exposure to grass may increase the odds. Additionally, PM2.5 may modify the associations of trees and grass with childhood asthma.

Building Facade Reconstruction Using Crowd- Sourced Photos and Two-Dimensional Maps

To address the high-cost problem of the current three-dimensional (&lt;small&gt;3D&lt;/small&gt;) reconstruction for urban buildings, a new technical framework is proposed to generate &lt;small&gt;3D&lt;/small&gt; building facade information using crowd-sourced photos and two-dimensional (2D) building vector data in this paper. The crowd-sourced photos mainly consisted of Tencent street view images and other-source photos, which were collected from three platforms, including search engines, social media, and mobile phones. The photos were selected and grouped first, and then a structure from motion algorithm was used for &lt;small&gt;3D&lt;/small&gt; reconstruction. Finally, the reconstructed point clouds were registered with 2D building vector data. The test implementation was conducted in the Jianye District of Nanjing, China, and the generated point clouds showed a good fit with the true values. The proposed &lt;small&gt;3D&lt;/small&gt; reconstruction method represents a multi-sourced data integration process. The advantage of the proposed approach lies in the open source and low-cost data used in this study.

Perceptions of built environment and health outcomes for older Chinese in Beijing: A big data approach with street view images and deep learning technique

Built environment attributes have been demonstrated to be associated with various health outcomes. However, most empirical studies have typically focused on objective built environmental measures. Still, perceptions of the built environment also play an important role in health and may complement studies with objective measures. Some built environment attributes, such as liveliness or beauty, are difficult to measure objectively. Traditional methods to assess perceptions of the built environment, such as questionnaires and focus groups, are time-consuming and prone to recall bias. The recent development in machine deep learning techniques and big data of street view images, makes it possible to assess perceptions of the built environment with street view images for a large-scale study area. By using online free Tencent Street View (TSV) images, this study assessed six perceptual attributes of the built environment: wealth, safety, liveliness, depression, bore and beauty. These attributes were associated with both the physical and the mental health outcomes of 1231 older adults in 48 neighborhoods in the Haidian District, Beijing, China. Results show that perceived safety was significantly associated with both the physical and mental health outcomes. Perceived depression and beauty were significant related to older adults' mental health, while perceived wealth, bore and liveliness were significantly related to their physical health. The findings carry important policy implications and hence contribute to the development of healthy cities. It is urgent to improve residents' positive perceptions and decrease their negative perceptions of the built environment, especially in neighborhoods that are highly populated by older adults.

The linkage between the perception of neighbourhood and physical activity in Guangzhou, China: using street view imagery with deep learning techniques

BackgroundNeighbourhood environment characteristics have been found to be associated with residents’ willingness to conduct physical activity (PA). Traditional methods to assess perceived neighbourhood environment characteristics are often subjective, costly, and time-consuming, and can be applied only on a small scale. Recent developments in deep learning algorithms and the recent availability of street view images enable researchers to assess multiple aspects of neighbourhood environment perceptions more efficiently on a large scale. This study aims to examine the relationship between each of six neighbourhood environment perceptual indicators—namely, wealthy, safe, lively, depressing, boring and beautiful—and residents’ time spent on PA in Guangzhou, China.MethodsA human–machine adversarial scoring system was developed to predict perceptions of neighbourhood environments based on Tencent Street View imagery and deep learning techniques. Image segmentation was conducted using a fully convolutional neural network (FCN-8s) and annotated ADE20k data. A human–machine adversarial scoring system was constructed based on a random forest model and image ratings by 30 volunteers. Multilevel linear regressions were used to examine the association between each of the six indicators and time spent on PA among 808 residents living in 35 neighbourhoods.ResultsTotal PA time was positively associated with the scores for “safe” [Coef. = 1.495, SE = 0.558], “lively” [1.635, 0.789] and “beautiful” [1.009, 0.404]. It was negatively associated with the scores for “depressing” [− 1.232, 0.588] and “boring” [− 1.227, 0.603]. No significant linkage was found between total PA time and the “wealthy” score. PA was further categorised into three intensity levels. More neighbourhood perceptual indicators were associated with higher intensity PA. The scores for “safe” and “depressing” were significantly related to all three intensity levels of PA.ConclusionsPeople living in perceived safe, lively and beautiful neighbourhoods were more likely to engage in PA, and people living in perceived boring and depressing neighbourhoods were less likely to engage in PA. Additionally, the relationship between neighbourhood perception and PA varies across different PA intensity levels. A combination of Tencent Street View imagery and deep learning techniques provides an accurate tool to automatically assess neighbourhood environment exposure for Chinese large cities.

The relationship between visual enclosure for neighbourhood street walkability and elders’ mental health in China: Using street view images

BackgroundNeighbourhood walkability has been consistently associated with more physical activities in adults. Nevertheless, evidence of a beneficial association between walkability and older adults’ mental health is still scarce in developing countries. Furthermore, walkability was often measured through Geographic Information System (GIS) methods. Recently, the methodological development in street view images, such as using Google or Tencent Street View, has offered an attractive alternative. MethodsUsing Tencent street view images and machine learning classification methods, we quantify visual enclosure for neighbourhood street walkability with the proportion of visible sky in those images. Depression and anxiety data were extracted from mental health surveys for 1231 older adults in Beijing, China in 2011. Multilevel linear regression models were used to assess the associations. ResultsStreet walkability was negatively associated with Geriatric Depression Scale scores (GDS 15-item) and Geriatric Anxiety Inventory scores (GAI 20-item), and the results of different robustness checks also support this relationship. The associations were stronger for disadvantaged older adults than others. ConclusionThe present study indicates a potential beneficial role of street walkability on mental health (depression and anxiety) in older residents, especially for disadvantaged older adults. Additional longitudinal studies are required to address some limitations of this study, such as residential self-selection bias.

Using deep learning to examine street view green and blue spaces and their associations with geriatric depression in Beijing, China

BackgroundResidential green and blue spaces may be therapeutic for the mental health. However, solid evidence on the linkage between exposure to green and blue spaces and mental health among the elderly in non-Western countries is scarce and limited to exposure metrics based on remote sensing images (i.e., land cover and vegetation indices). Such overhead-view measures may fail to capture how people perceive the environment on the site. ObjectiveThis study aimed to compare streetscape metrics derived from street view images with satellite-derived ones for the assessment of green and blue space; and to examine associations between exposure to green and blue spaces as well as geriatric depression in Beijing, China. MethodsQuestionnaire data on 1190 participants aged 60 or above were analyzed cross-sectionally. Depressive symptoms were assessed through the shortened Geriatric Depression Scale (GDS-15). Streetscape green and blue spaces were extracted from Tencent Street View data by a fully convolutional neural network. Indicators derived from street view images were compared with a satellite-based normalized difference vegetation index (NDVI), a normalized difference water index (NDWI), and those derived from GlobeLand30 land cover data on a neighborhood level. Multilevel regressions with neighborhood-level random effects were fitted to assess correlations between GDS-15 scores and these green and blue spaces exposure metrics. ResultsThe average cumulative GDS-15 score was 3.4 (i.e., no depressive symptoms). Metrics of green and blue space derived from street view images were not correlated with satellite-based ones. While NDVI was highly correlated with GlobeLand30 green space, NDWI was moderately correlated with GlobeLand30 blue space. Multilevel regressions showed that both street view green and blue spaces were inversely associated with GDS-15 scores and achieved the highest model goodness-of-fit. No significant associations were found with NDVI, NDWI, and GlobeLand30 green and blue space. Our results passed robustness tests. ConclusionOur findings provide support that street view green and blue spaces are protective against depression for the elderly in China, yet longitudinal confirmation to infer causality is necessary. Street view and satellite-derived green and blue space measures represent different aspects of natural environments. Both street view data and deep learning are valuable tools for automated environmental exposure assessments for health-related studies.

Measuring visual quality of street space and its temporal variation: Methodology and its application in the Hutong area in Beijing

Although it is widely known that quality of street space plays a vital role in promoting urban vibrancy, there is still no consensus on how to quantitatively measure it for a large scale. Recent emerging dataset Street View Picture has revealed the possibility to overcome the previous limit, thus bringing forward a research paradigm shift. Taking this advantage, this paper explores a new approach for visual quality evaluation and variation identification of street space for a large area. Hutongs, which typically represent for historical street space in Beijing, are selected for empirical study. In the experimental part, we capture multi-years Tencent Street View Picture covering all the Hutongs, and conduct both physical and perceived visual quality evaluation. The physical visual quality of street space is achieved automatically by combining 3-dimensional composition calculation of greenery, openness, enclosure using machine-learning segmentation method SegNet, and 2-dimensional analysis of street wall continuity and cross-sectional proportion; perceived visual quality of street space is evaluated by stay willingness scoring from five aspects. The variation of quality is evaluated based on the identified physical space variations. The result indicates that visual quality of Hutongs are not satisfied, while some regeneration projects in the historical protection block is better. Most Hutongs are in shortage of visual green, relative more continuous but with low cross-sectional ratio. Hutongs near main road witness an increasing trend of motorization. The difference between physical and perceived quality indicates the feasibility and limitation of the auto-calculation method. In the most recent 3–4 years, less than 2.5% Hutongs are improved, which are mainly slow beautification.

Traffic signal detection and classification in street views using an attention model

Detecting small objects is a challenging task. We focus on a special case: the detection and classification of traffic signals in street views. We present a novel framework that utilizes a visual attention model to make detection more efficient, without loss of accuracy, and which generalizes. The attention model is designed to generate a small set of candidate regions at a suitable scale so that small targets can be better located and classified. In order to evaluate our method in the context of traffic signal detection, we have built a traffic light benchmark with over 15,000 traffic light instances, based on Tencent street view panoramas. We have tested our method both on the dataset we have built and the Tsinghua–Tencent 100K (TT100K) traffic sign benchmark. Experiments show that our method has superior detection performance and is quicker than the general faster RCNN object detection framework on both datasets. It is competitive with state-of-the-art specialist traffic sign detectors on TT100K, but is an order of magnitude faster. To show generality, we tested it on the LISA dataset without tuning, and obtained an average precision in excess of 90%.

How Green Are the Streets Within the Sixth Ring Road of Beijing? An Analysis Based on Tencent Street View Pictures and the Green View Index.

Street greenery, an important urban landscape component, is closely related to people’s physical and mental health. This study employs the green view index (GVI) as a quantitative indicator to evaluate visual greenery from a pedestrian’s perspective and uses an image segmentation method to calculate the quantity of visual greenery from Tencent street view pictures. This article aims to quantify street greenery in the area within the sixth ring road in Beijing, analyse the relations between road parameters and the GVI, and compare the visual greenery of different road types. The authors find that (1) the average GVI value in the study area is low, with low-value clusters inside the third ring road and high-value clusters outside; (2) wider minor roads tend to have higher GVI values than motorways, major roads and provincial roads; and (3) longer roads, except expressways, tend to have higher GVI values. This case study demonstrates that the GVI can effectively represent the quantity of visual greenery along roads. The authors’ methods can be employed to compare street-level visual greenery among different areas or road types and to support urban green space planning and management.

Use of Tencent Street View Imagery for Visual Perception of Streets

The visual perception of streets plays an important role in urban planning, and contributes to the quality of residents’ lives. However, evaluation of the visual perception of streetscapes has been restricted by inadequate techniques and the availability of data sources. The emergence of street view services (Google Street View, Tencent Street View, etc.) has provided an enormous number of new images at street level, thus shattering the restrictions imposed by the limited availability of data sources for evaluating streetscapes. This study explored the possibility of analyzing the visual perception of an urban street based on Tencent Street View images, and led to the proposal of four indices for characterizing the visual perception of streets: salient region saturation, visual entropy, a green view index, and a sky-openness index. We selected the Jianye District of Nanjing City, China, as the study area, where Tencent Street View is available. The results of this experiment indicated that the four indices proposed in this work can effectively reflect the visual attributes of streets. Thus, the proposed indices could facilitate the assessment of urban landscapes based on visual perception. In summary, this study suggests a new type of data for landscape study, and provides a technique for automatic information acquisition to determine the visual perception of streets.

Automatic Sky View Factor Estimation from Street View Photographs—A Big Data Approach

Hemispherical (fisheye) photography is a well-established approach for estimating the sky view factor (SVF). High-resolution urban models from LiDAR and oblique airborne photogrammetry can provide continuous SVF estimates over a large urban area, but such data are not always available and are difficult to acquire. Street view panoramas have become widely available in urban areas worldwide: Google Street View (GSV) maintains a global network of panoramas excluding China and several other countries; Baidu Street View (BSV) and Tencent Street View (TSV) focus their panorama acquisition efforts within China, and have covered hundreds of cities therein. In this paper, we approach this issue from a big data perspective by presenting and validating a method for automatic estimation of SVF from massive amounts of street view photographs. Comparisons were made with SVF estimates derived from two independent sources: a LiDAR-based Digital Surface Model (DSM) and an oblique airborne photogrammetry-based 3D city model (OAP3D), resulting in a correlation coefficient of 0.863 and 0.987, respectively. The comparisons demonstrated the capacity of the proposed method to provide reliable SVF estimates. Additionally, we present an application of the proposed method with about 12,000 GSV panoramas to characterize the spatial distribution of SVF over Manhattan Island in New York City. Although this is a proof-of-concept study, it has shown the potential of the proposed approach to assist urban climate and urban planning research. However, further development is needed before this approach can be finally delivered to the urban climate and urban planning communities for practical applications.

How green are the streets? An analysis for central areas of Chinese cities using Tencent Street View.

Extensive evidence has revealed that street greenery, as a quality-of-life component, is important for oxygen production, pollutant absorption, and urban heat island effect mitigation. Determining how green our streets are has always been difficult given the time and money consumed using conventional methods. This study proposes an automatic method using an emerging online street-view service to address this issue. This method was used to analyze street greenery in the central areas (28.3 km2 each) of 245 major Chinese cities; this differs from previous studies, which have investigated small areas in a given city. Such a city-system-level study enabled us to detect potential universal laws governing street greenery as well as the impact factors. We collected over one million Tencent Street View pictures and calculated the green view index for each picture. We found the following rules: (1) longer streets in more economically developed and highly administrated cities tended to be greener; (2) cities in western China tend to have greener streets; and (3) the aggregated green view indices at the municipal level match with the approved National Garden Cities of China. These findings can prove useful for drafting more appropriate policies regarding planning and engineering practices for street greenery.