The COVID-19 pandemic continues to significantly impact people’s lives worldwide, emphasizing the critical need for effective detection methods. Many existing deep learning-based approaches for COVID-19 detection offer high accuracy but demand substantial computing resources, time, and energy. In this study, we introduce an optical diffractive neural network (ODNN-COVID), which is characterized by low power consumption, efficient parallelization, and fast computing speed for COVID-19 detection. In addition, we explore how the physical parameters of ODNN-COVID affect its diagnostic performance. We identify the F number as a key parameter for evaluating the overall detection capabilities. Through an assessment of the connectivity of the diffractive network, we established an optimized range of F number, offering guidance for constructing optical diffractive neural networks. In the numerical simulations, a three-layer system achieves an impressive overall accuracy of 92.64% and 88.89% in binary- and three-classification diagnostic tasks. For a single-layer system, the simulation accuracy of 84.17% and the experimental accuracy of 80.83% can be obtained with the same configuration for the binary-classification task, and the simulation accuracy is 80.19% and the experimental accuracy is 74.44% for the three-classification task. Both simulations and experiments validate that the proposed optical diffractive neural network serves as a passive optical processor for effective COVID-19 diagnosis, featuring low power consumption, high parallelization, and fast computing capabilities. Furthermore, ODNN-COVID exhibits versatility, making it adaptable to various image analysis and object classification tasks related to medical fields owing to its general architecture.