KRAB-containing zinc finger proteins (KZNF) constitute the largest family of transcriptional regulators in humans and play critical roles in normal development and tumorigenesis. However, the function and mechanism of most KZNFs remain unclear. Here, we report that ZNF496, a KZNF family member, interacts with the DNA binding domain (DBD) of estrogen receptor alpha (ERα) via its C2H2 domain. This interaction decreases ERα binding to chromatin DNA and results in the repression of ERα transactivation, the selective suppression of ERα target genes, and ultimately in a reduction of ERα-positive cell growth in the presence of E2. An analysis of clinical data revealed that the downregulation of ZNF496 expression is observed only in ERα-positive and not in ERα-negative breast cancer tissues when compared with that in matched adjacent tissues. Lastly, we also observed that the downregulation of ZNF496 is associated with poor recurrence-free survival among patients with breast cancer. Collectively, our findings demonstrate that ZNF496 is a novel ERα-binding protein that acts as a target gene-specific ERα corepressor and inhibits the growth of ERα-positive breast cancer cells.