In this paper we develop an estimation method for extracting non-affine latent stochastic volatility and risk premia from measures of model-free realized and risk-neutral integrated volatility. We estimate non-affine models with nonlinear drift and constant elasticity of variance and we compare them to the popular square-root stochastic volatility model. Our empirical findings are: (1) the square-root model is misspecified; (2) the inclusion of constant elasticity of variance and nonlinear drift captures stylized facts of volatility dynamics; (3) models with linear drift imply an explosive volatility process under the risk-neutral probability measure. To gauge the economic impact of the empirical findings we also study the implications of non-affine specifications on dynamic asset allocation strategies with stochastic volatility. We show that, in contrast to the affine case, non-affine volatility models induce market timing and can generate volatility dynamics which are volatile enough to produce large intertemporal hedging demands.