Crowd flow prediction has been studied for a variety of purposes, ranging from the private sector such as location selection of stores according to the characteristics of commercial districts and customer-tailored marketing to the public sector for social infrastructure design such as transportation networks. Its importance is even greater in light of the spread of contagious diseases such as COVID-19. In many cases, crowd flow can be divided into subgroups by common characteristics such as gender, age, location type, etc. If we use such hierarchical structure of the data effectively, we can improve prediction accuracy of crowd flow for subgroups. But the existing prediction models do not consider such hierarchical structure of the data. In this study, we propose a deep learning model based on global–local structure of the crowd flow data, which utilizes the overall(global) and subdivided by the types of sites(local) crowd flow data simultaneously to predict the crowd flow of each subgroup. The experiment result shows that the proposed model improves the prediction accuracy of each sub-divided subgroup by 5.2% (Table 5 Cat #9)—45.95% (Table 11 Cat #5), depending on the data set. This result comes from the comparison with the related works under the same condition that use target category data to predict each subgroup. In addition, when we refine the global data composition by considering the correlation between subgroups and excluding low correlated subgroups, the prediction accuracy is further improved by 5.6–48.65%.