Using a new specification of multifactor volatility, we estimate the hidden risk factors spanning S&P 500 index (SPX) implied volatility surfaces and the risk premia of volatility-sensitive payoffs. SPX implied volatility surfaces are well-explained by three dependent state variables reflecting (i) short- and long-term implied volatility risks and (ii) short-term implied skewness risk. The more persistent volatility factor and the skewness factor support a downward sloping term structure of variance risk premia in normal times, whereas the most transient volatility factor accounts for an upward sloping term structure in periods of distress. Our volatility specification based on a matrix state process is instrumental to obtaining a tractable and flexible model for the joint dynamics of returns and volatilities, which improves pricing performance and risk premium modeling with respect to recent three-factor specifications based on standard state spaces. This paper was accepted by Gustavo Manso, finance.