AbstractIn this paper, conventional phonon blockade (CPNB) and conventional photon blockade (CPTB) effects, as well as unconventional phonon blockade (UPB) effects, are studied in an optomechanical system with nonlinear interaction between the cavity frequency and the square of the mechanical displacement driven by an external field, where a two‐level atom couples with the mechanical mode and a microwave driving field pumps cavity mode. The second‐order correlation function is analytically calculated, which is in good agreement with the numerical simulation given by the master equation. With energy‐level diagram, the atom‐mechanical mode coupling is found to induces the degeneracy splitting of the states and give the optimal conditions for CPNB and CPTB in this system. With the origin of UPB, the optimal conditions are derived and it is found that the realization of UPB is determined by the two couplings of the cavity and atom with respect to the mechanical mode. Moreover, some discussions on the experimental implementation in this quadratically coupled optomechanical system are presented. This study provides a possible way for realizing single‐photon nonlinearity and can extend the applications of optomechanical systems in the field of quantum optics.