Building change detection from very-high-resolution (VHR) urban remote sensing image frequently encounter the challenge of serious false alarm caused by different illumination or viewing angles in bi-temporal images. An approach to alleviate the false alarm in urban building change detection is proposed in this paper. Firstly, as shadows casted by urban buildings are of distinct spectral and shape feature, it adopts a supervised object-based classification technique to extract them in this paper. Secondly, on the opposite direction of sunlight illumination, a straight line is drawn along the principal orientation of building in every extracted shadow region. Starting from the straight line and moving toward the sunlight direction, a rectangular area is constructed to cover partial shadow and rooftop of each building. Thirdly, an algebra and geometry invariant based method is used to abstract the spatial topological relationship of the potential unchanged buildings from all central points of the rectangular area. Finally, based on an oriented texture curvature descriptor, an index is established to determine the actual false alarm in building change detection result. The experiment results validate that the proposed method can be used as an effective framework to alleviate the false alarm in building change detection from urban VHR image.