We consider a predator–prey system with prey population guided anti-predator behavior, in which anti-predator behaviors happen only when the population size of the prey is greater than a threshold. We investigate the rich dynamics of the proposed piecewise model as well as both subsystems without and with nonlinear functional response. In particular, the subsystem with anti-predator behaviors exhibits rich dynamical behaviors including saddle-node bifurcation, Hopf bifurcation, Bogdanov–Takens bifurcation and homoclinic bifurcation. Further, besides the dynamical properties of subsystems the piecewise system shows some new complicated dynamical behaviors as the threshold value varies, including unstable limit cycle, semistable limit cycle, bistability of equilibrium and limit cycle, and tristability of three equilibria. From the switching system we can conclude that a great anti-predator rate induces the prey population to persist more likely, but whether the prey and predator populations coexist depends further on the threshold that triggers anti-predator behavior. Especially, a large threshold not only makes coexistence of the prey and predator populations as an equilibrium more likely, but also damps the predator–prey oscillations.