What deep learning can (and can’t) see: a systemic review assessing applicability for quantifying qualities of urban vitality

The learning process and hyper-parameter optimization of artificial neural networks (ANNs) and deep learning (DL) architectures is considered one of the most challenging machine learning problems. Several past studies have used gradient-based back propagation methods to train DL architectures. However, gradient-based methods have major drawbacks such as stucking at local minimums in multi-objective cost functions, expensive execution time due to calculating gradient information with thousands of iterations and needing the cost functions to be continuous. Since training the ANNs and DLs is an NP-hard optimization problem, their structure and parameters optimization using the meta-heuristic (MH) algorithms has been considerably raised. MH algorithms can accurately formulate the optimal estimation of DL components (such as hyper-parameter, weights, number of layers, number of neurons, learning rate, etc.). This paper provides a comprehensive review of the optimization of ANNs and DLs using MH algorithms. In this paper, we have reviewed the latest developments in the use of MH algorithms in the DL and ANN methods, presented their disadvantages and advantages, and pointed out some research directions to fill the gaps between MHs and DL methods. Moreover, it has been explained that the evolutionary hybrid architecture still has limited applicability in the literature. Also, this paper classifies the latest MH algorithms in the literature to demonstrate their effectiveness in DL and ANN training for various applications. Most researchers tend to extend novel hybrid algorithms by combining MHs to optimize the hyper-parameters of DLs and ANNs. The development of hybrid MHs helps improving algorithms performance and capable of solving complex optimization problems. In general, the optimal performance of the MHs should be able to achieve a suitable trade-off between exploration and exploitation features. Hence, this paper tries to summarize various MH algorithms in terms of the convergence trend, exploration, exploitation, and the ability to avoid local minima. The integration of MH with DLs is expected to accelerate the training process in the coming few years. However, relevant publications in this way are still rare.

Classification of urban morphology with deep learning: Application on urban vitality

The six dimensions of built environment on urban vitality: Fusion evidence from multi-source data

Urban vitality is a key indicator for measuring urban development. This topic has been trending in urban planning and sustainable development, and significant progress has been made in measuring single indicators of urban vitality based on parcel or block units. With the continuous development of smart sensing technology, multisource urban data are becoming increasingly abundant. The application of such data to measure the multidimensional urban vitality of street space, reflecting multiple functions of an urban space, can significantly improve the accuracy of urban vitality analyses and promote the construction of people-oriented healthy cities. In this study, streets were taken as the analysis unit, and multisource data such as the trajectories of taxies and shared bicycles, user reviews and cultural facility points of interest (POIs) in Chengdu, a city in southwestern China, were used to identify spatial patterns of urban vitality on streets across social, economic and cultural dimensions. The correlation between the built environment factors and the multidimensional urban vitality on the street was analyzed using a multiple regression model. The spatial distribution of the different dimensions of urban vitality of the street space in Chengdu varies to a certain extent. It is common for areas with high social vitality to have production and life centers nearby. High economic vitality centers are typically found along busy streets with a high concentration of businesses. Areas with high cultural vitality centers tend to be concentrated on the city’s central streets. Land use, transportation, external environment, population and employment are all closely linked to urban vitality on streets. The crowd counting and POI density have the greatest impact on multidimensional urban vitality. The crowd and the level of service facilities profoundly affect social interaction, trade activities and cultural communication. The goodness of fit (R2) of the regression models for social, economic and cultural vitality are 0.590, 0.423 and 0.409, respectively. Using multisource urban data, our findings can help stakeholders better understand the spatial patterns and influencing factors of multidimensional urban vitality on streets and provide sustainable urban planning and development strategies for the future.

The automatic segmentation of brain tumour from MRI medical images is mainly covered in this review. Recently, state-of-the-art performance is provided by deep learning-based approaches in the field of image classification, segmentation, object detection, and tracking tasks. The core feature deep learning approach is the hierarchical representation of features from images and thus avoiding domain-specific handcrafted features. In this review paper, we have dealt with a Review of Deep Learning Architecture and Methods for MRI Brain Tumour Segmentation. First, we have discussed basic architecture and approaches for deep learning methods. Secondly, we have discussed the literature survey of MRI brain tumour segmentation using deep learning methods and its multimodality fusion. Then, the advantages and disadvantages of each method analyzed and finally concluded the discussion with the merits and challenges of deep learning techniques. The review of brain tumour identification using deep learning Techniques may help the research to have a better focus on it.

Urban vitality reflects the dynamism, openness, available resources, and linkages between different elements in a city. Many studies have focused on the relationship between urban vitality and the built environment, which can benefit urban planners. Based on multi-source data, this study incorporates the aspects of economy, society, culture, and innovation to comprehensively measure urban vitality. Recognizing the significant influence of the built environment on urban vitality, this paper takes Nanjing as the study area. It explores the relationship between urban vitality and the built environment using ordinary least squares and multi-scale geographically weighted regression models based on multi-source data. The results reveal multi-centered urban vitality within Nanjing’s central urban area, which gradually decreases outwards from the city center. Particularly, the southern region has a higher comprehensive vitality than the northern region. While differences in vitality between dimensions are evident, an overall consistent pattern emerges. The spatial distribution of influence varies across different spatial factor. The method provided in this study gives a new view of urban vitality measurement and evaluation.

Detecting changes in land use and land cover (LULC) from space has long been the main goal of satellite remote sensing (RS), yet the existing and available algorithms for cloud classification are not reliable enough to attain this goal in an automated fashion. Clouds are very strong optical signals that dominate the results of change detection if they are not removed completely from imagery. As various architectures of deep learning (DL) have been proposed and advanced quickly, their potential in perceptual tasks has been widely accepted and successfully applied to many fields. A comprehensive survey of DL in RS has been reviewed, and the RS community has been suggested to be leading researchers in DL. Based on deep residual learning, semantic image segmentation, and the concept of atrous convolution, we propose a new DL architecture, named CloudNet, with an enhanced capability of feature extraction for classifying cloud and haze from Sentinel-2 imagery, with the intention of supporting automatic change detection in LULC. To ensure the quality of the training dataset, scene classification maps of Taiwan processed by Sen2cor were visually examined and edited, resulting in a total of 12,769 sub-images with a standard size of 224 × 224 pixels, cut from the Sen2cor-corrected images and compiled in a trainset. The data augmentation technique enabled CloudNet to have stable cirrus identification capability without extensive training data. Compared to the traditional method and other DL methods, CloudNet had higher accuracy in cloud and haze classification, as well as better performance in cirrus cloud recognition. CloudNet will be incorporated into the Open Access Satellite Image Service to facilitate change detection by using Sentinel-2 imagery on a regular and automatic basis.

Parkinson's disease (PD) is a globally significant health challenge, necessitating accurate and timely diagnostic methods to facilitate effective treatment and intervention. In recent years, self-supervised deep representation pattern learning (SS-DRPL) has emerged as a promising approach for extracting valuable representations from data, offering the potential to enhance the efficiency of voice-based PD detection. This research study focuses on investigating the utilization of SS-DRPL in conjunction with deep learning algorithms for voice-based PD classification. This study encompasses a comprehensive evaluation aimed at assessing the accuracy of various predictive models, particularly deep learning methods when combined with SS-DRPL. Two deep learning architectures, namely hybrid Long Short-Term Memory and Recurrent Neural Networks (LSTM-RNN) and Deep Neural Networks (DNN), are employed and compared in terms of their ability to detect voice-based PD cases accurately. Additionally, several traditional machine learning models are also included to establish a baseline for comparison. The findings of the study reveal that the incorporation of SS-DRPL leads to improved model performance across all experimental setups. Notably, the LSTM-RNN architecture augmented with SS-DRPL achieves the highest F1-score of 0.94, indicating its superior ability to detect PD cases using voice-based data effectively. This outcome underscores the efficacy of SS-DRPL in enabling deep learning models to learn intricate patterns and correlations within the data, thereby facilitating more accurate PD classification.

Optimizing the built environment to foster urban vitality is essential for effective urban planning and sustainable development. Previous studies have predominantly focused on analyzing the relationship between the built environment and urban vitality at either a macro or micro-scale, often assuming a predefined linear relationship. In this study, we investigate the potential non-linear interactions between the built environment and urban vitality by employing an interpretable spatial machine learning framework that integrates the XGBoost model with the SHapley Additive exPlanations (SHAP) algorithm. Additionally, we analyze the determinants of urban vitality across both micro and macro-scales using multi-source data, semantic segmentation models, and street view imagery. Our findings reveal the following key insights: (1) the distribution of urban vitality exhibits spatial heterogeneity within the main urban area of Shanghai, with high vitality areas concentrated in the Huangpu District and at intersections with neighboring districts, demonstrating a decline from the center to the periphery; (2) the XGBoost model outperforms other comparative models, showcasing superior capabilities in simulating and predicting urban vitality; (3) among the various built environment factors influencing urban vitality, building coverage, population density, and distance to the CBD exert the most significant effects, while the green view index and the number of bus stops contribute relatively less; (4) all built environment factors demonstrate nonlinear impacts and exhibit certain threshold effects on urban vitality. The analytical outcomes of this study provide valuable insights for optimizing the spatial layout and resource allocation within urban settings, offering references for urban planning and sustainable development initiatives.

Deep learning methods have been applied when working to enhance the prediction accuracyof traditional statistical methods in the field of plant breeding. Although deep learningseems to be a promising approach for genomic prediction, it has proven to have somelimitations, since its conventional methods fail to leverage all available information.Multimodal deep learning methods aim to improve the predictive power of their unimodalcounterparts by introducing several modalities (sources) of input information. In thisreview, we introduce some theoretical basic concepts of multimodal deep learning andprovide a list of the most widely used neural network architectures in deep learning, aswell as the available strategies to fuse data from different modalities. We mention someof the available computational resources for the practical implementation of multimodaldeep learning problems. We finally performed a review of applications of multimodal deeplearning to genomic selection in plant breeding and other related fields. We present ameta-picture of the practical performance of multimodal deep learning methods to highlighthow these tools can help address complex problems in the field of plant breeding. Wediscussed some relevant considerations that researchers should keep in mind when applyingmultimodal deep learning methods. Multimodal deep learning holds significant potential forvarious fields, including genomic selection. While multimodal deep learning displaysenhanced prediction capabilities over unimodal deep learning and other machine learningmethods, it demands more computational resources. Multimodal deep learning effectivelycaptures intermodal interactions, especially when integrating data from different sources.To apply multimodal deep learning in genomic selection, suitable architectures and fusionstrategies must be chosen. It is relevant to keep in mind that multimodal deep learning,like unimodal deep learning, is a powerful tool but should be carefully applied. Given itspredictive edge over traditional methods, multimodal deep learning is valuable inaddressing challenges in plant breeding and food security amid a growing globalpopulation.

Identifying urban vitality in large cities is critical for optimizing the urban fabric. While great attention has been paid to urban vitality in developed countries, related studies have been rarely conducted in developing countries. In this study, we defined urban vitality as the capacity of an urban built environment to boost lively social activities and developed a framework for measuring urban vitality using the dimensions of built environment, human activities, and human–environment interaction. Taking Shanghai, China as a case, we conducted a measurement of urban vitality using multi-source data. The results show that Shanghai follows a monocentric vital pattern within the outer ring road, with urban vitality declining from the central urban core to the city periphery. While the old urban cores tend to show high urban vitality, Pudong New Area is mostly dominated by low vitality. Three clusters with high urban vitality were identified: the old urban area, the Lujiazui CBD, and residential agglomeration areas. We conducted validation of the measuring results using phone usage density. Urban vitality showed a positive correlation with phone usage density, indicating a high accuracy of assessment. We also discovered that European-style block planning, zoning plan, mixed-functional development, urban renewal regulation, and migrant concentration were playing leading role in urban vitality of Shanghai.

Abstract. 3D building reconstruction using Earth Observation (EO) data (aerial and satellite imagery, point clouds, etc.) is an important and active research topic in different fields, such as photogrammetry, remote sensing, computer vision and Geographic Information Systems (GIS). Nowadays 3D city models have become an essential part of 3D GIS environments and they can be used in many applications and analyses in urban areas. The conventional 3D building reconstruction methods depend heavily on the data quality and source; and manual efforts are still needed for generating the object models. Several tasks in photogrammetry and remote sensing have been revolutionized by using deep learning (DL) methods, such as image segmentation, classification, and 3D reconstruction. In this study, we provide a review on the state-of-the-art machine learning and in particular the DL methods for 3D building reconstruction for the purpose of city modelling using EO data. This is the first review with a focus on object model generation based on the DL methods and EO data. A brief overview of the recent building reconstruction studies with DL is also given. We have investigated the different DL architectures, such as convolutional neural networks (CNNs), generative adversarial networks (GANs), and the combinations of conventional approaches with DL in this paper and reported their advantages and disadvantages. An outlook on the future developments of 3D building modelling based on DL is also presented.

Accurate solar radiation forecasts are necessary for solar power plants and local energy grids. Precise solar predictions can optimise plant operations and assist in managing power generation fluctuations. High-quality solar data are required to perform solar forecasting and may be accessed through open databases such as the Baseline Surface Radiation Network. Due to maintenance and malfunctioning equipment, these databases may have missing data. Deep learning methods can extract relevant features and non-linearities in data, which make them suitable for forecasting and imputation tasks. The development of the transformer architecture in 2017 marked an improvement from recurrent sequence processing using RNN (recurrent neural net) architectures to parallel sequence handling using self-attention mechanisms. Previous deep learning irradiance forecasting models utilised a RNN or RNN extension architecture. This paper presents novel applications of Transformer models to solar imputation and forecasting. Two deep learning architectures – Transformer and SAITS (Self-Attention-based Imputation for Time Series) – were trained to impute time series of global horizontal irradiance. Additionally, a Temporal Fusion Transformer (TFT) deep learning architecture was trained to forecast future values of global horizontal irradiance time series. Feature concatenation allows for the combination of differing exogenous variables into a single input vector for a deep learning solar forecasting model. Previous solar forecasting models do not incorporate cloud data or otherwise utilise an exogenous simple categorical mask to indicate cloud cover. In this paper, GOES-16 cloud optical depth and particle size datasets provide an improved representation of cloud properties. Classical imputation and forecasting models were also developed and used as baselines for comparison with deep learning models. The models were evaluated using standard statistical metrics for time series. It was found that the deep learning TFT forecasting model produced better forecasts than the classical regression model. However, the classical KNN imputer performed better than the two deep learning imputation models.

Sensing urban vitality is a useful method for understanding urban development. However, the spatio-temporal characteristics of the association between the built environment and urban vitality in Shenzhen, the youngest mega-city in China, have not yet been explored. In this paper, we examined the effects of built environment indicators on urban vitality by using spatial regression models and multi-source geospatial data. The main research findings were as follows. Firstly, urban vitality displayed a consistent high–low pattern during both weekdays and weekends. Differences in the distribution of urban vitality with time between weekdays and weekends were more significant. Secondly, the effects of various built environment indicators on urban vitality exhibited significant temporal disparities. Within a day, population density, building density, bus station density, and distance to metro stations all exhibited positive effects, while distance to the central business district (CBD) exhibited negative effects, with pronounced diurnal differences. Moreover, the effects of road network density and functional mix on urban vitality were both positive and negative throughout the day. Thirdly, besides population density and building density, road network density, functional mix, bus stop density, and distance from metro stations exhibited positive and negative disparities within the study area. Overall, distance to the CBD had a negative effect on urban vitality. This concludes that planning for urban vitality should consider the spatio-temporal heterogeneity of the association between the built environment and urban vitality.

A quantitative study of urban vitality brings new insights for evaluating the external construction environment and internal development power of cities. However, it still has limited knowledge of the relations between people’s diverse urban life and urban vitality, although urban activities are often used as the proxy for urban vitality. This paper aims to deeply mine the content of urban social life and reveal the driving mechanism of urban vitality after inspecting human activities. We propose a general framework for exploring the spatial pattern and driving mechanism of urban vitality using multi-source big data. It builds a mapping relationship between various urban activities and urban vitality aspects, including economic and social. In addition, the physical environment (static) and human–land interaction (dynamic) indicators are designed to analyze the driving mechanism of urban vitality using the Geographically Weighted Regression model. The results show that the spatial pattern and driving factors of urban vitality are heterogeneous over space regarding both the economic and social aspects of our experimental study. This work provides us with multiple perspectives to understand the connotation of urban vitality and urges us to develop rational strategies to make the city more vital, coordinated, and sustainable.