We apply a natural language processing algorithm to FOMC statements to construct a new measure of monetary policy stance, including the tone and novelty of a policy statement. We exploit cross-sectional variations across alternative FOMC statements to identify the tone (for example, dovish or hawkish), and contrast the current and previous FOMC statements released after Committee meetings to identify the novelty of the announcement. We then use high-frequency bond prices to compute the surprise component of the monetary policy stance. Our text-based estimates of monetary policy surprises are not sensitive to the choice of bond maturities used in estimation, are highly correlated with forward guidance shocks in the literature, and are associated with lower stock returns after unexpected policy tightening. The key advantage of our approach is that we are able to conduct a counterfactual policy evaluation by replacing the released statement with an alternative statement, allowing us to perform a more detailed investigation at the sentence and paragraph level.