A post-disaster scene is more complicated than a vertical pre-disaster one, since it includes debris-covered areas and the imagery angle may be oblique. Detecting buildings and their irregularities in post-disaster oblique images is a valuable resource in many remote sensing applications, such as building damage assessment, which is a critical and challenging task in the disaster management cycle. We proposed a two-level fusion method, which is fast enough to support save and rescue missions. Firstly, an edge-based knowledge-based approach is designed to extract the vertical building map from temporal pre-disaster data. This approach utilizes Gray-Level Co-Occurrence Matrix (GLCM) features and shadow information. The temporal analysis helps the procedure to find stable objects by reducing the effect of noise in fine-resolution data. The mentioned two-level fusion includes data and decision levels. Spectral and georeferenced bands of pre- and post-disaster samples are fused in the data level. The decision-level refers to fusing the results of two classifiers, Spectral-only and Geo-Spectral classifiers. The locality of the method is preserved by the georeferenced feature. The proposed method is implemented on the Google Earth Engine (GEE) platform, and the evaluation is based on Hurricane Nate (2017) and Hurricane Harvey (2017) oblique images. The proposed method shows a significant reduction in the false-positive error, and provides a high performance in detecting irregularity in facade and rooftop areas.