Visual Fire Detection (VFD), through the rapid and accurate identification of smoke and flame in images and videos, is crucial for early fire warning and reducing fire hazards. In recent years, the introduction of deep learning has significantly advanced this field, especially in the automatic extraction of discriminative features necessary for VFD. This paper provides a comprehensive review of the latest technological advancements in fire detection using deep learning, offering a broad perspective. Initially, it details the publicly available benchmark datasets widely used in VFD research and the corresponding evaluation metrics, providing a basis for researchers to assess the performance of various algorithms. Subsequently, we propose a systematic categorization framework, dividing VFD tasks into three key directions: fire classification, fire localization, and fire segmentation. For these directions, we thoroughly review the innovative improvements in deep learning models tailored for image and video inputs and discusses how these advancements enhance the accuracy and efficiency of fire detection. Finally, we highlight the challenges in the field and explore future research directions, intending to inspire and guide both newcomers and seasoned researchers in this area.