Genetic polymorphisms in select genes, including APOE (apolipoprotein E), COMT (Catechol-O-Methyltransferase), MDR1 (multi-drug resistance 1), BDNF (brain derived neurotrophic factor), and GST (glutathione-S-transferase), have been associated with vulnerability to cognitive impairment. In this study, we evaluated the relationship of these genetic variants to measures of brain health in patients with breast cancer, including neurocognitive testing and functional connectome analysis. Women with breast cancer (n = 83) and female healthy controls (n = 53) were evaluated. They underwent resting-state functional MRI scans and neurocognitive testing. Polymerase chain reaction (PCR) was performed on saliva samples to identify single nucleotide polymorphisms (SNPs) in candidate genes: APOE, COMT, MDR1, BDNF, and GST. Breast cancer patients treated with chemotherapy had slower processing speed (p = 0.04) and poorer reported executive function (p < 0.0001) than healthy controls. Those chemotherapy-treated patients that were APOE e4 carriers had significantly slower processing speed. A greater number of risk-related alleles was associated with poorer connectivity in the regions of the left cuneus and left calcarine. While breast cancer patients that are APOE e4 carriers may have a select vulnerability to processing speed impairments, other risk-related alleles were not found to influence cognitive test performance in this population. Conversely, regions of impaired functional connectivity appeared to be related to risk-related genetic polymorphisms in breast cancer patients. This suggests that a cancer patient’s SNPs in candidate genes may influence the risk of neurotoxicity. Further study evaluating the impact of genotype on biomarkers of brain health in cancer survivors is warranted.