Substantial technological development has made Unmanned Aerial Vehicles (UAVs) more versatile, cheaper and accessible to the public in recent years. Alongside many positive effects and use cases, safety concerns are increasing as a plethora of incidents demonstrate the destructive potential of UAVs. To counteract this development and thus protect people and critical infrastructure, UAV detection, tracking and defence gains more and more research attention. Whereas, different drone detection technologies like RADAR, radio frequency and acoustic detection are deployed within multi-spectral systems, optical detection and imaging of approaching objects provide key information to correctly assess the situation. As reaction time is a crucial parameter for successful UAV defence, the operating distance of the optical detection system needs to be improved further. This paper presents the analysis, development and evaluation of a telescope-based UAV detection system. The system consists of a high precision mount and a telescope equipped with a camera. UAVs are detected in the captured video frames by the deep learning algorithm YOLOv4 using a modified architecture. The proposed system, which uses a f/10 telescope with a focal length of f = 2540 mm and a camera equipped with a 7.3 mm x 4.1 mm sensor, allows a significant increase of the optical detection range to more than 3 km of UAVs down to 0.3 m in diameter under daylight conditions and sufficient contrast, extending the reaction time significantly for counter UAV systems.