The aim of this paper is to apply the modified potential well method and some new differential inequalities to study the asymptotic behavior of solutions to the initial homogeneous Neumann problem of a nonlinear diffusion equation driven by the p(x)-Laplace operator. Complete classification of global existence and blow-up in finite time of solutions is given when the initial data satisfies different conditions. We first obtain a threshold result for the solution to exist globally or to blow up in finite time when the initial energy is subcritical or critical. We also establish a decay rate of the L2 norm for global solutions. Furthermore, we provide some sufficient conditions for the existence of global and blow-up solutions for supercritical initial energy. We finally give two-sided estimates of asymptotic behavior when the diffusion term dominates the source. This is a continuation of our previous work [17].