Remote sensing (RS) image change detection analyzes and determines the difference of surface states at different times. Polygons are common vector data to express the surface states in geographic information systems. However, the production and updating of polygons typically depend on manual procedures, which costs significant time and human resources. This article proposes an automatic polygon change detection method based on high-resolution RS images and deep learning image classification. First, bitemporal images are segmented by boundary-preserved masking simple linear iterative clustering based on the boundaries and categories of polygons. Then, the labeled dataset is automatically generated by adaptively cropping image segments according to the center-to-boundary distance. Date-1 data are used to train the model and purified by two different classifiers to improve the model performance. Date-2 data, whose categories are predicted, are the basic units of change detection analysis. Furthermore, an improved bilinear convolutional neural network with a reduced number of parameters and training time is applied to fine-grained classification. Finally, the changed polygons can be retrieved based on category comparison and post-processing. In our experiments, two real datasets with high-resolution RS images and land-use polygons are employed to validate the effectiveness of our proposed method. The results are examined by visual interpretation and show that the precision and recall rates for dataset 1 are 90.5% and 93.1%, respectively, while those for dataset 2 are 87.6% and 90.8%. Through our method, the existing polygons can be used appropriately, and the difficulty of data updating can be reduced.