Strategies to prevent the growth of mycotoxin-producing fungi in agricultural products need to be identified. In this study, the antifungal efficacy of Perilla frutescens essential oil (PEO) as a fumigant against Aspergillus flavus (A. flavus) in stored rice seeds was investigated. PEO affected the permeability and integrity of the A. flavus cell membrane, as evidenced by the decreased cell viability. In situ assays demonstrated that fumigation with PEO significantly reduced and retarded the growth of A. flavus in stored rice seeds but the germination rate decreased with increasing concentration of PEO. PEO could alleviate rice seeds’ color differences by reducing the degree of rice seeds browning, as evidenced by a decrease in polyphenol oxidase in PEO-fumigated rice seeds. Moreover, the rice seeds with PEO fumigation exhibited higher soluble protein content and catalase (CAT) activity, which collectively suggests that PEO promotes the tolerance of rice seeds to A. flavus. Furthermore, decreases in fatty acid value (FAV), malondialdehyde (MDA) content, and acceptable sensorial characteristics were observed in the rice seeds after PEO fumigation, indicating that fumigation with PEO can effectively delay the trend of quality deterioration in high moisture rice seeds during storage. These results recommended PEO as a potential green preservative for the shelf-life extension of stored grain.