Selenocystine (SeC) has been identified as a novel compound with broad-spectrum anticancer activity. However, the effects of SeC on modifying DNA repair mechanism were less addressed. In this study, we demonstrated that SeC selectively induced cytotoxicity and genotoxicity against HepG2 hepatoma cell line. Comet assay revealed SeC-induced DNA damage in HepG2 cells, particularly in the form of DNA double strand breaks (DSBs), corroborated by the increase expression of the DSB marker, gamma-H2AX. We further demonstrated that SeC suppressed DNA homologous recombination repair, exacerbating DNA damage accumulation. Such effects on DNA damage and cell viability inhibition were alleviated by antioxidants, glutathione and Trolox, suggesting the involvement of reactive oxygen species (ROS). High levels of intracellular and mitochondrial ROS were detected in SeC-treated HepG2. In addition, SeC impaired the expression of antioxidant enzymes (superoxidase mutases and catalase), prompting the imbalance between antioxidant protection and excessive ROS formation and eliciting DSBs and cellular death. Decreased procaspase-3, 7, and 9 and Bcl-2 proteins and an increased Bax/Bcl-2 ratio, were observed after SeC treatment, but could be reversed by Torlox, confirming the action of SeC on ROS-induced apoptosis. In vivo, the xenograft tumor model of HepG2 cells validated the inhibition of SeC on tumor growth, and the induction of DSBs and apoptosis.In summary, SeC has the capability to induce ROS-dependent DNA damage and impeded DBS repair in HepG2 cells. Thus, SeC holds great promise as a therapeutic or adjuvant agent targeting DNA repair for cancer treatment.