Annihilator varieties of distinguished modules of reductive Lie algebras

Abstract

AbstractWe provide a microlocal necessary condition for distinction of admissible representations of real reductive groups in the context of spherical pairs.Let${\mathbf {G}}$be a complex algebraic reductive group and${\mathbf {H}}\subset {\mathbf {G}}$be a spherical algebraic subgroup. Let${\mathfrak {g}},{\mathfrak {h}}$denote the Lie algebras of${\mathbf {G}}$and${\mathbf {H}}$, and let${\mathfrak {h}}^{\bot }$denote the orthogonal complement to${\mathfrak {h}}$in${\mathfrak {g}}^*$. A${\mathfrak {g}}$-module is called${\mathfrak {h}}$-distinguished if it admits a nonzero${\mathfrak {h}}$-invariant functional. We show that the maximal${\mathbf {G}}$-orbit in the annihilator variety of any irreducible${\mathfrak {h}}$-distinguished${\mathfrak {g}}$-module intersects${\mathfrak {h}}^{\bot }$. This generalises a result of Vogan [Vog91].We apply this to Casselman–Wallach representations of real reductive groups to obtain information on branching problems, translation functors and Jacquet modules. Further, we prove in many cases that – as suggested by [Pra19, Question 1] – whenHis a symmetric subgroup of a real reductive groupG, the existence of a temperedH-distinguished representation ofGimplies the existence of a genericH-distinguished representation ofG.Many of the models studied in the theory of automorphic forms involve an additive character on the unipotent radical of the subgroup$\bf H$, and we have devised a twisted version of our theorem that yields necessary conditions for the existence of those mixed models. Our method of proof here is inspired by the theory of modules overW-algebras. As an application of our theorem we derive necessary conditions for the existence of Rankin–Selberg, Bessel, Klyachko and Shalika models. Our results are compatible with the recent Gan–Gross–Prasad conjectures for nongeneric representations [GGP20].Finally, we provide more general results that ease the sphericity assumption on the subgroups, and apply them to local theta correspondence in type II and to degenerate Whittaker models.