Direction-of-arrival (DoA) estimation is one of the most promising technologies in array signal processing. Existing data-driven methods for DoA estimation are usually implemented by classification networks, which suffer from insufficient utilization about features of sources and require spectral peak-search stage. In this letter, we reframe DoA estimation as a target detection problem and propose a novel DoA estimation approach on the basis of the you only look once v3 (YOLOv3) framework, namely YOLO-DoA. DoAs of sources with confidence scores are directly predicted from the spectrum proxy with YOLO-DoA and an end-to-end estimation is realized. By combining squeeze-and-excitation operation, cross stage partial connections, and an improved loss function for bounding box regression, the performance of YOLO-DoA is enhanced. Simulation results demonstrate that the proposed approach outperforms several state-of-the-art methods in terms of network size, computational cost, prediction time, and accuracy of DoA estimation.