Academic health centers (AHCs) face unique challenges in providing continuity to a medically and socially complex patient population. Little is known about what drives patient loss in these settings. Determine physician- and patient-based factors associated with patient loss in AHCs. Retrospective cohort study, embedded qualitative analysis. Academic health center. All visits from 7/1/2014 to 6/30/2019; 89 physicians (51%) participated in a qualitative analysis. Physician-based factors (gender, years of service, hours of practice per week, trainee status, and departure during the study period) and patient-based factors (age, gender, race, limited English proficiency, public health insurance, chronic illness burden, and severe psychiatric illness burden) and their association with patient loss to follow-up, defined as a lapse in provider visit greater than 3years. We identified 402,415 visits for 41,876 distinct patients. A total of 9332 (22.3%) patients were lost to follow-up. Patient factors associated with loss to follow-up included patient age < 40 (HR 3.12 (2.94-3.33)), identification as non-white (HR 1.07 (1.10-1.13)), limited English proficiency (HR 1.18 (1.04-1.33)), and use of public insurance (HR 1.12 (1.04-1.21)). Provider factors associated with patient loss included trainee status (HR 3.74 (2.43-5.75)) and having recently departed from the practice (HR 1.98, 1.66-2.35). Structured interviews with clinical providers revealed unfavorable relationships with providers and staff (35%), inconvenience accessing primary care (23%), unreliable health insurance (18%), difficulty accessing one's primary care provider (14%), and patient/provider transitions (10%) as reasons for patient loss. Younger patient age, markers of social vulnerability, and physician transiency are associated with patient loss at AHCs, providing targets to improve continuity of care within these settings.