CrowdOptim: A Crowd-driven Neural Network Hyperparameter Optimization Approach to AI-based Smart Urban Sensing

Abstract

AI-based smart urban sensing (ASUS) has emerged as a scalable and pervasive application paradigm in smart city planning and management that aims to automatically assess the physical status of the urban environments by leveraging AI techniques and massive urban sensing data. In this paper, we focus on a crowd-driven neural network (NN) hyperparameter optimization problem in ASUS applications. Our goal is to utilize the human intelligence from crowdsourcing systems to identify the optimal NN hyperparameter configuration for an ASUS model. Our work is motivated by the observation that the hyperparameters of current ASUS models are often manually configured by the AI specialists, which is known to be both error-prone and suboptimal. Two key technical challenges exist in solving our problem: i) it is challenging to effectively translate the highly complex NN hyperparameter optimization problem in AI to a simplified problem that can be solved by crowd workers without extensive AI expertise; ii) it is difficult to identify the optimal hyperparameter configuration in the large hyperparameter search space given the blackbox nature of the AI model. To address the above challenges, we develop CrowdOptim, a crowd-AI collaborative learning framework that integrates the techniques from crowdsourcing, hyperparameter optimization, and estimation theory to address the crowd-driven NN hyperparameter optimization problem in ASUS applications. The evaluation results from two real-world ASUS applications (i.e., smart city infrastructure monitoring (SCIM) and urban environment cleanliness assessment (UECA)) show that CrowdOptim consistently outperforms the state-of-the-art deep convolutional networks, crowd-AI, and hyperparameter optimization baselines in achieving the ASUS application objectives under various evaluation scenarios.