Little is understood of the consequences of restrictive visitor policies that were implemented in hospitals to minimize risk of infection during the COVID-19 pandemic. The objective of this study was to describe physician experiences with these policies and reflections of their effects. We conducted semistructured phone interviews from September 2020 to March 2021 with physicians practising in Ontario hospitals, recruited via professional networks and snowball sampling. We audio-recorded, transcribed and analyzed interviews to describe and interpret overarching themes by thematic analysis. We interviewed 21 physicians (5 intensivists, 5 internists, 11 specialists in palliative care). Four main thematic categories emerged, including provider, system, patient and caregiver effects. Provider-related factors included increased time and effort on communication with a need to establish limits; increased effort to develop rapport with caregivers; lack of caregiver input on patient care; the need to act as a caregiver surrogate; and the emotional toll of being a gatekeeper or advocate for visitors, exacerbated by lack of evidence for restrictions and inconsistent enforcement. System effects included the avoidance of hospital admission and decreased length of stay, leading to readmissions, increased deaths at home and avoidance of transfer to other facilities with similar policies. Patient-related factors included isolation and dying alone; lack of caregiver advocacy; and prioritization of visitor presence that, at times, resulted in a delay or withdrawal of aspects of care. Caregiver-related factors included inability to personally assess patient health, leading to poor understanding of patient status and challenging decision-making; perceived inadequate communication; difficulty accessing caregiver supports; and increased risk of complicated grief. Participants highlighted a disproportionate effect on older adults and people who did not speak English. Our study highlights substantial negative consequences of restrictive visitor policies, with heightened effects on older adults and people who did not speak English. Research is required to identify whether the benefits of visitor restrictions on infection control outweigh the numerous deleterious consequences to patients, families and care providers.