We trace the evolution and the star formation history of passive galaxies, using a subset of the VIMOS Public Extragalactic Redshift Survey (VIPERS). We extracted from the VIPERS survey a sample of passive galaxies in the redshift range 0.4<z<1.0 and stellar mass range 10<$log(M_{star}/M_{\odot})$<12. The sample was selected using an evolving cut in the rest-frame U-V color distribution and additional quality-ensuring cuts. We use the stacked spectra to measure the 4000$\AA$ break (D4000) and the $H\delta$ Lick index ($H\delta_{A}$) with high precision. We compare the results with a grid of synthetic spectra to constrain the star formation epochs of these galaxies. We characterize the formation redshift-stellar mass relation for intermediate-redshift passive galaxies. We find that at $z\sim1$ stellar populations in low-mass passive galaxies are younger than in high-mass passive galaxies, similarly to what is observed at the present epoch. Over the full analyzed redshift and stellar mass range, the $D4000$ index increases with redshift, while $H\delta_{A}$ gets lower. This implies that the stellar populations are getting older with increasing stellar mass. Comparison to the spectra of passive galaxies in the SDSS survey shows that the shape of the relations of $D4000$, and $H\delta_{A}$ with stellar mass has not changed significantly with redshift. Assuming a single burst formation, this implies that high-mass passive galaxies formed their stars at $z_{form}\sim2$, while low-mass galaxies formed their main stellar population more recently, at $z_{form}\sim1$. The consistency of these results, obtained using two independent estimator of the formation redshift ($D4000$ and $H\delta_{A}$), further strengthens a scenario in which star formation proceeds from higher- to lower-mass systems as time passes, i.e. what has become known as the 'downsizing' picture.