PurposeVisitor restriction policies to prevent the spread of COVID-19 among patients and clinicians were widespread during the pandemic, resulting in the exclusion of caregivers at key points of cancer care and treatment decision-making. The aim of this study was to explore how visitor restrictions impacted cancer treatment decision-making and care from patient and physician perspectives.MethodsSixty-seven interviews, including 48 cancer patients and 19 cancer and palliative care physicians from four academic cancer centers in the USA between August 2020 and July 2021.ResultsVisitor restrictions that prevented caregivers from participating in clinic appointments and perioperative hospital care created challenges in cancer care that spanned three domains: practical, social, and informational. We identified eight themes that characterized challenges within the three domains across all three groups, and that these challenges had negative emotional and psychological consequences for both groups. Physicians perceived that patients’ negative experiences due to lack of support through the physical presence of caregivers may have worsened patient outcomes.ConclusionsOur data demonstrate the tripartite structure of the therapeutic relationship in cancer care with caregivers providing critical support in the decision-making and care process to both patients and physicians. Caregiver absences led to practical, psychosocial, and informational burdens on both groups, and likely increased the risk of burnout among physicians. Our findings suggest that the quality of cancer care can be enhanced by engaging caregivers and promoting their physical presence during clinical encounters.