We consider theories of three dimensional quantum gravity in Anti-de Sitter space which possess massless higher-spin gauge symmetry. The perturbative spectrum of the theory includes higher spin excitations which can be organized into vacuum representations of the W_N algebra; these are higher spin versions of the boundary gravitons. We describe a fundamental bound which relates the value of the cosmological constant to the amount of gauge symmetry present. In the dual CFT language, this is the statement that modular invariance implies that the theory can not be quantized unless the central charge is sufficiently large, i.e. if c is greater than or equal to N-1. This bound relies on the assumption that all of the perturbative excitations exist as full states in the quantum theory, and can be circumvented if the theory possesses a linearization instability. The W_N minimal models -- recently conjectured to be dual to certain higher spin AdS theories by Gaberdiel and Gopakumar - provide an example of this phenomenon. This result can be regarded as an example of a "gravitational exclusion principle" in Anti-de Sitter space, where a non-perturbative quantum gravity mechanism involving black holes places a limit on the number of light degrees of freedom present.