Integral formulas with distribution kernels for irreducible projections in L2 of a nilmanifold

Abstract

Let N be a simply connected nilpotent Lie group and Γ a discrete uniform subgroup. The authors consider irreducible representations σ in the spectrum of the quasi-regular representation N × L 2( Γ/ N) → L 2( Γ→) which are induced from normal maximal subordinate subgroups M ⊆ N. The primary projection P σ and all irreducible projections P ⩽ P σ are given by convolutions involving right Γ-invariant distributions D on Γ→, Pf(Γn) = D ∗ f(Γn) = <D, n · f> all f ϵ C ∞(Γ/N) , where n · f( ζ) = f( ζ · n). Extending earlier work of Auslander and Brezin, and L. Richardson, the authors give explicit character formulas for the distributions, interpreting them as sums of characters on the torus T κ = ( Γ ∩ M) · [ M, M]⧹ M. By examining these structural formulas, they obtain fairly sharp estimates on the order of the distributions: if σ is associated with an orbit O ⊆ n ∗ and if V ⊆ n ∗ is the largest subspace which saturates θ in the sense that f ϵ O ⇒ f + V ⊆ O . As a corollary they obtain Richardson's criterion for a projection to map C 0( Γ→) into itself. The authors also resolve a conjecture of Brezin, proving a Zero-One law which says, among other things, that if the primary projection P σ maps C r ( Γ→) into C 0( Γ→), so do all irreducible projections P ⩽ P σ . This proof is based on a classical lemma on the extent to which integral points on a polynomial graph in R n lie in the coset ring of Z n (the finitely additive Boolean algebra generated by cosets of subgroups in Z n ). This lemma may be useful in other investigations of nilmanifolds.