Transfer operators and Hankel transforms between relative trace formulas, II: Rankin–Selberg theory

Abstract

The Langlands functoriality conjecture, as reformulated in the “beyond endoscopy” program, predicts comparisons between the (stable) trace formulas of different groups G1,G2 for every morphism G1L→LG2 between their L-groups. This conjecture can be seen as a special case of a more general conjecture, which replaces reductive groups by spherical varieties and the trace formula by its generalization, the relative trace formula.The goal of this article and its precursor [11] is to demonstrate, by example, the existence of “transfer operators” between relative trace formulas, which generalize the scalar transfer factors of endoscopy. These transfer operators have all properties that one could expect from a trace formula comparison: matching, fundamental lemma for the Hecke algebra, transfer of (relative) characters. Most importantly, and quite surprisingly, they appear to be of abelian nature (at least, in the low-rank examples considered in this paper), even though they encompass functoriality relations of non-abelian harmonic analysis. Thus, they are amenable to application of the Poisson summation formula in order to perform the global comparison. Moreover, we show that these abelian transforms have some structure — which presently escapes our understanding in its entirety — as deformations of well-understood operators when the spaces under consideration are replaced by their “asymptotic cones”.In this second paper we use Rankin–Selberg theory to prove the local transfer behind Rudnick's 1990 thesis (comparing the stable trace formula for SL2 with the Kuznetsov formula) and Venkatesh's 2002 thesis (providing a “beyond endoscopy” proof of functorial transfer from tori to GL2). As it turns out, the latter is not completely disjoint from endoscopic transfer — in fact, our proof “factors” through endoscopic transfer. We also study the functional equation of the symmetric-square L-function for GL2, and show that it is governed by an explicit “Hankel operator” at the level of the Kuznetsov formula, which is also of abelian nature. A similar theory for the standard L-function was previously developed (in a different language) by Jacquet.