The fluid inerter described by the fractional derivative model is introduced into the traditional nonlinear energy sink (NES), which is called fractional-order NES in this paper. The slowly varying dynamic equation (SVDE) of the system coupled with fractional-order NES is obtained by the complex averaging method, in which the fractional derivative term is treated using the fractional Leibniz theorem. Then, the discriminants (Δ, Δ1, and Δ2) of the number of equilibrium points are derived. By using the variable substitution method, the characteristic equation for judging the stability is established. The results show: (1) the approximate SVDE is sufficient to reflect the slowly varying characteristics of the system, which shows that the mathematical treatment of the fractional derivative term is reliable; (2) the discriminant conditions (Δ1, Δ2) can accurately reflect the number of equilibrium points, and the corresponding range of nonlinear parameter κ can be calculated when the system has three equilibrium points. The expressions of Δ1, Δ2 are simpler than Δ, which is suitable for analysis and design parameters; (3) the stability discrimination methods of schemes 1 and 2 are accurate. Compared with scheme 2, scheme 1 is more prone to various responses, especially various strongly and weakly modulated responses. In scheme 2, the inertia effect of mass can be completely replaced by integer order inerter. Compared with integer order inerter, the introduction of fractional order inerter, whether in series or in parallel, means that the amplitude of the equilibrium point on the NES vibrator is smaller, but it is also for this reason that it is not easy to produce a modulated response with scheme 2, and the vibration suppression effect of the main structure is not good.