The extraction of building outline vectors is an essential task in supporting various applications. Although the recent development of deep learning based techniques has made advancements in the automation of this task, the accuracy and precision are insufficient due to errors caused by abundant noise and obstruction around buildings in aerial images. To better address this issue, this paper presents a new approach called the multi-task edge detection (MTED) for building vectorization with the following characteristics. First, instead of detecting building corner points that are very sensitive to noise effects, a deep learning based rotated bounding box detector is introduced for building edge detection to increase robustness to interference. Second, a multi-task learning strategy is designed to integrate building segmentation inside the METD framework to closely guide the edge detection using spatial context. Third, a simple yet effective geometry-guided post-processing method is designed to reconstruct vectorized building outlines based on the detected edges and learned building shape prior knowledge. The comparative experiments conducted on benchmark very-high-resolution optical aerial images indicate that the proposed approach can significantly outperform the state-of-the-arts in terms of vertex-based building outline accuracy metrics. With a test time of 58ms per building, this method enables efficient building polygon labeling in interactive mapping applications for building surveying and mapping. Code is available at https: //github.com/yifanthomaswu/MTED_framework.