On the spectral theory of groups of affine transformations of compact nilmanifolds

Abstract

Let N be a connected and simply connected nilpotent Lie group, � a lattice in N, and �\N the corresponding nilmanifold. Let Aff(�\N) be the group of affine transformations of �\N. We characterize the countable subgroups H of Aff(�\N) for which the action of H on �\N has a spectral gap, that is, such that the associated unitary representation U 0 of H on the space of functions from L 2 (�\N) with zero mean does not weakly contain the trivial representation. Denote by T the maximal torus factor associated to �\N. We show that the action of H on �\N has a spectral gap if and only if there exists no proper H-invariant subtorus S of T such that the projection of H on Aut(T/S) has an abelian subgroup of finite index. We first establish the result in the case where �\N is a torus. In the case of a general nilmanifold, we study the asymptotic behaviour of matrix coefficients ofU 0 using decay properties of metaplectic representations of symplectic groups. The result shows that the existence of a spectral gap for subgroups of Aff(�\N) is equivalent to strong ergodicity in the sense of K. Schmidt. Moreover, we show that the action of H on �\N is ergodic (or strongly mixing) if and only if the corresponding action of H on T is ergodic (or strongly mixing).