Early recognition of citrus diseases is important for preventing crop losses and employing timely disease control measures in farms. Employing machine learning-based approaches, such as deep learning for accurate detection of multiple citrus diseases is challenging due to the limited availability of labeled diseased samples. Further, a lightweight architecture with low computational complexity is required to perform citrus disease classification on resource-constrained devices, such as mobile phones. This enables practical utility of the architecture to perform effective monitoring of diseases by farmers using their own mobile devices in the farms. Hence, we propose a lightweight, fast, and accurate deep metric learning-based architecture for citrus disease detection from sparse data. In particular, we propose a patch-based classification network that comprises an embedding module, a cluster prototype module, and a simple neural network classifier, to detect the citrus diseases accurately. Evaluation of our proposed approach using publicly available citrus fruits and leaves dataset reveals its efficiency in accurately detecting the various diseases from leaf images. Further, the generalization capability of our approach is demonstrated using another dataset, namely the tea leaves dataset. Comparison analysis of our approach with existing state-of-the-art algorithms demonstrate its superiority in terms of detection accuracy (95.04%), the number of parameters required for tuning (less than 2.3 M), and the time efficiency in detecting the citrus diseases (less than 10 ms) using the trained model. Moreover, the ability to learn with fewer resources and without compromising accuracy empowers the practical utility of the proposed scheme on resource-constrained devices, such as mobile phones.