In this paper, direct numerical simulations (DNSs) are performed to investigate the deformation and breakup of an elastoviscoplastic (EVP) droplet in a Newtonian matrix under simple shear flow. The two-phase interface is captured by the volume-of-fluid (VOF) method with adaptive mesh refinement technique. The Saramito model (Bingham model coupled exponential Phan-Thien–Tanner viscoelastic model) is used to characterize the rheological behavior of the droplet. The droplet deformation and conformational state are studied with different Capillary numbers Ca, Weissenberg numbers Wi, and Bingham numbers Bi, which represent the surface tension, elasticity, and yield stress of the droplet, respectively. Our results show that droplet deformation occurs at low Ca, while breakup occurs at high Ca. The droplet non-monotonically deforms with increasing Wi and Bi, while is elongated for higher Ca. In addition, three breakup modes (mid-point pinching, transitional breakup, and homogeneous breakup) are reported for EVP droplets, in which transitional breakup disappears due to the influence of high elasticity. The conformational state of the droplet intuitively demonstrates the change of breakup from horizontal shear to vertical breakup. In spite of the fact that the surface tension always inhibits the deformation of droplets, the present work indicates that Bi has little effect on the deformation with high Wi and high Ca, while the influence is obvious at low Wi and Ca. The observed elastic and plastic effects on droplet deformation and breakup are believed to have significant impacts, as yield stress fluids are widely encountered in industrial applications.