Background: Endometriosis affects approximately 4%–10% of women of reproductive age. The most common symptom is dysmenorrhea, which can impact patients’ physical, mental, and social lives. Early diagnosis is imperative, despite the fact that it is indeed challenging that diagnostic delay occurs frequently. Both physician and patient factors contribute to the diagnostic delay of endometriosis. Methods: We conducted a literature search published from March 2015 to March 2023 in PubMed, Cochrane, PMC, CINAHL, and SCOPUS databases reporting about factors associated with delayed diagnosis of endometriosis. Results: Six studies were included in the analysis. Patient aspects, including age at first symptom ( p < 0.001) and first consultation ( p < 0.001), were associated with delayed diagnosis of endometriosis. Physician aspects, which include the number of doctors’ visits to get a diagnosis ( p < 0.001), time to referral to a gynecologist ( p = 0.028), and false diagnosis by a general practitioner ( p = 0.021), are also associated with delayed diagnosis and treatment of endometriosis. From patient’s perspectives, their symptoms are often normalized by general practitioners, and general practitioners lack knowledge regarding endometriosis. From a health professional’s perspective, time-consuming consultation correlates with delayed diagnosis. Conclusion: Factors associated with delayed diagnosis of endometriosis were patient—and physician-related, including age at first symptom and first consultation, normalization of symptoms, and lack of knowledge among general practitioners. Increasing awareness and knowledge about endometriosis will likely reduce the diagnostic delay.