With the rapid shift to telehealth, there remains a knowledge gap in how video-based care is implemented in interdisciplinary primary care (PC) settings. The objective of this study was to gain an in-depth understanding of how video telehealth services were implemented in PC from the perspectives of patients and interdisciplinary PC team members at the Veterans Health Administration (VHA) 2 years after the onset of the COVID-19 pandemic. We applied a positive and negative deviance approach and selected the 6% highest (n=8) and the 6% lowest (n=8) video-using PC sites in 2022 from a total of 130 VHA medical centers nationally. A total of 12 VHA sites were included in the study, where 43 PC interdisciplinary team members (August-October 2022) and 25 patients (February-May 2023) were interviewed. The 5 domains from the diffusion of innovation theory and the nonadoption, abandonment, scale-up, spread, and sustainability (NASSS) framework guided the development of the 2 study interview guides (provider and patient). We identified themes that emerged across all interviews that were associated with the implementation of video-based care in interdisciplinary PC settings, using directed-content rapid analysis of the interview transcripts. The analysis was guided by 5 a priori NASSS domains: (1) patient condition or characteristic, (2) technology, (3) adopter system, (4) health care organization, and (5) adaptation over time. The study findings include the following common themes and factors, organized by the 5 NASSS domains: (1) patient condition or characteristic-visit type or purpose (eg, follow-up visits that do not require physical examination), health condition (eg, homebound or semihomebound patients), and sociodemographic characteristic (eg, patients who have a long commute time); (2) technology-key features (eg, access to video-enabled devices), knowledge (eg, how to use videoconferencing software), and technical support for patients and providers; (3) adopter system-changes in staff roles and clinical practice (eg, coordination of video-based care), provider and patient preference or comfort to use video-based care, and caregiver's role (eg, participation of caregivers during video visits); (4) health care organization-leadership support and access to resources, scheduling for video visits (eg, schedule or block off digital half or full days), and training and telehealth champions (eg, hands-on or on-site training for staff, patients, or caregivers); (5) adaptation over time-capacity to improve all aspects of video-based care and provide continued access to resources (eg, effective communication about updates). This study identified key factors associated with the implementation of video-based services in interdisciplinary PC settings at the VHA from the perspectives of PC team members and patients. The identified multifaceted factors may inform recommendations on how to sustain and improve the provision of video-based care in VHA PC settings as well as non-VHA patient-centered medical homes.