The continuous spectrum of the relative trace formula for GL(3) over a quadratic extension

Abstract

LetE/F be a quadratic extension of number fields,G the group GL(3,E) regarded as an algebraic group overF andU a quasi-split unitary group in three variables. Let alsoϑ be a generic character of a maximal unipotent subgroupN ofG. We derive an explicit expression for the integral $$\int {\int {K_{cont} (u, n)du\theta (n)dn} } $$ whereK cont is the continuous part of the kernel attached to a smooth function of compact support onG(A). In particular, we prove that this expression is absolutely convergent. The result can be used to show that a cuspidal representation ofG contains a vectorφ such thateφ(u)du≠0 if and only if it is a base change from a representation of GL(3,F).