Abstract After the occurrence of various types of disasters, including natural disasters and man-made damage, aid workers need accurate and timely data, such as the damage status of buildings, in order to take effective measures for rescue. So as to solve this problem, this paper researches and designs a building damage classification system based on machine learning. The damage assessment system consists of two network models (building extraction network and damage classification network). This article analyzes and designs the structure of each network model, and discusses the principles related to computer vision in machine learning. Buildings in satellite images are segmented through Siamese Convolutional Neural Network, the BottleNeck Module and Feature Pyramid Network are used in the damage classification assessment network to detect damage to buildings in sub-temporal remote sensing images. Subsequently, the model was trained and tested on different disaster events on the xBD dataset. The results show that the building damage detection system based on Siamese-CNN achieves good detection accuracy, and the system has the advantages of simple operation, good timeliness and low resource consumption, and can well meet the needs of disaster assessment.