We present a study of the evolution of the galaxy Velocity Dispersion Function (VDF) from z=0 to z=1.5 using photometric data from the UKIDSS Ultra Deep Survey (UDS) and Newfirm Medium Band Survey (NMBS) COSMOS surveys. The VDF has been measured locally using direct kinematic measurements from the Sloan Digital Sky Survey, but direct studies of the VDF at high redshift are difficult as they require velocity dispersion measurements of many thousands of galaxies. Taylor et al. (2010) demonstrated that dynamical and stellar mass are linearly related when the structure of the galaxy is accounted for. We show that the stellar mass, size and S\'ersic index can reliably predict the velocity dispersions of SDSS galaxies. We apply this relation to galaxies at high redshift and determine the evolution of the inferred VDF. We find that the VDF at z~0.5 is very similar to the VDF at z=0. At higher redshifts, we find that the number density of galaxies with dispersions <~200 km/s is lower, but the number of high dispersion galaxies is constant or even higher. At fixed cumulative number density, the velocity dispersions of galaxies with logN[Mpc^{-3}]<-3.5 increase with time by a factor of ~1.4 from z~1.5-0, whereas the dispersions of galaxies with lower number density are approximately constant or decrease with time. The VDF appears to show less evolution than the stellar mass function, particularly at the lowest number densities. We note that these results are still somewhat uncertain and we suggest several avenues for further calibrating the inferred velocity dispersions.