Treatment modifications and contact restrictions were common during the COVID-19 pandemic and can be stressors for mental health. There is a lack of studies assessing pandemic-related risk factors for anxiety and depression of cancer patients and survivors systematically in multifactorial models. A total of 2391 participants, mean age 65.5 years, ≤5 years post-diagnosis of either lung, prostate, breast, colorectal cancer, or leukemia/lymphoma, were recruited in 2021 via the Baden-Württemberg Cancer Registry, Germany. Sociodemographic information, pandemic-related treatment modifications, contact restrictions, and anxiety/depression (Hospital Anxiety and Depression Scale, HADS) were assessed via self-administered questionnaire. Clinical information (diagnosis, stage, and treatment information) was obtained from the cancer registry. Overall, 22% of participants reported oncological care modifications due to COVID-19, mostly in follow-up care and rehabilitation. Modifications of active cancer treatment were reported by 5.8%. Among those, 50.5% had subclinical anxiety and 55.4% subclinical depression (vs. 37.4% and 45.4%, respectively, for unchanged active treatment). Age <60 years, female sex, lung cancer, low income, and contact restrictions to peer support groups or physicians were identified as independent risk factors for anxiety. Risk factors for depression were lung cancer (both sexes), leukemia/lymphoma (females), recurrence or palliative treatment, living alone, low income, and contact restrictions to relatives, physicians, or caregivers. The study demonstrates that changes in active cancer treatment and contact restrictions are associated with impaired mental well-being. The psychological consequences of treatment changes and the importance for cancer patients to maintain regular contact with their physicians should be considered in future responses to threats to public health.