The role of punishment in the evolution of cooperation has been disputed for a long time because its effectiveness is challenged by antisocial punishment. Here we introduce neutral punishment, which is executed only if personal payoff is less than the average payoff of neighbors, into spatial prisoner’s dilemma game. In addition, we consider memory length to control the time span of recovery. It is shown that cooperative behavior is remarkably promoted and even dominant with increment of punishment rate and memory length in spite of the high value of temptation. Moreover, we draw a conclusion that this facilitation phenomenon of cooperation is due to heterogeneity of individual fitness coefficient, thus verifying the positive effect of heterogeneity on the evolution of cooperation through network reciprocity.