Cytoplasmic male sterility (CMS) plays a crucial role in the utilization of heterosis. The petaloid anther abortion in oil tea (Camellia oleifera Abel.) constitutes a CMS phenomenon, which is of great value for the hybrid breeding of oil tea. However, as the mechanism of its CMS is still poorly understood, it is necessary to study the cytology and physiological characteristics of anther abortion. In this study, a C. oleifera cultivar, Huashuo (HS), and its petalized CMS mutant (HSP) were used as materials to explore this mechanism. Compared with HS, cytological analysis demonstrated that HSP showed early-onset tapetum programmed cell death (PCD) and an organelle disorder phenotype during the tetrad stage. In HSP, anthers exhibited elevated levels of calcium deposition in anther wall tissues, tapetum layers, and microspores, and yet calcium accumulation was abnormal at the later stage. The contents of hydrogen peroxide and MDA in HSP anthers were higher, and the activities of superoxide dismutase (SOD), peroxidase (POD), and catalase (CAT) were lower than those of HS, which resulted in an excessive accumulation of reactive oxygen species (ROS). Real-time quantitative PCR confirmed that the transcription levels of CoPOD and CoCAT genes encoding key antioxidant enzymes in HSP were downregulated compared with HS in early pollen development; the gene CoCPK, which encodes a calcium-dependent protein kinase associated with antioxidase, was upregulated during the critical period. Thus, we suggest that excessive ROS as a signal breaks the balance of the antioxidant system, and along with an abnormal distribution of calcium ions, leads to the early initiation of PCD in the tapetum, and ultimately leads to pollen abortion for HSP. These results lay a cytological and physiological foundation for further studies on the CMS mechanism, and provide information for breeding male-sterile lines of C. oleifera.