In this paper, two non-linear hypotheses are tested on the controversial time-series relationship between investor sentiment and market returns: i) an interaction, subject to abrupt regime shifts, and ii) a gradual sentiment effect, which alters the influences of other factors, such as the volatility premium, as a sentiment threshold is exceeded. Both hypotheses are supported by the data (vs. the corresponding linear alternatives) for the SP500 index and institutional (but not individual) sentiment over the period 1965-2003, and after controlling for various risk factors. A mutual influence, significant both in statistical and economic terms, exists between monthly returns and institutional sentiment, during a dominant market regime with occurrence probability 80%. Instead, individual sentiment exerts no significant effect on SP500 returns, although it responds positively to them. Institutional and individual investors are influenced by each others' sentiment, but they interpret these as opposite signals, contrarian and momentum respectively. Similarly, they perceive past volatility as a source of optimism/pessimism. Interestingly, aggregate idiosyncratic volatility, a proxy for total arbitrage cost, exerts a positive impact on both subsequent returns and institutional sentiment, indicating that institutions correctly predict higher returns as this cost increases (e.g. due to an anticipated correction of a mispricing) or possibly, that they partially contribute to this pattern via their own trading. A smooth-transition regression specification reveals that, in a similar way that sentiment alters, at the individual stock level, the effects of firm characteristics on returns (Baker and Wurgler, 2006), institutional sentiment alters, at the market level, the sign and magnitude of the volatility effect. This indicates a compensation for sentiment risk, as implied by De Long et al. (1990). Accounting for regime shifts seems critical for return prediction over month-ahead horizons.