Counterexamples to Analytic Hypoellipticity for Domains of Finite Type

Abstract

Consider the hypersurface {(Z1, Z2): Q(Z2) = P(zi)} cC C2, where P : R is a subharmonic, nonharmonic polynomial. Such a surface is pseudoconvex (more precisely, is the boundary of a pseudoconvex domain) and of finite type. It is the boundary of a model domain {f(Z2) > P(zi)}, so called because such domains provide good approximations to general pseudoconvex domains of finite type in C2 (see [37],[11]). A nonvanishing, antiholomorphic, tangent vector field is a/Oaz-1 -2i(aP/Oaz-i)O/a29z-. As coordinates for the surface we use C x JR 3 (z, t) -(z, t + iP(z)); the vector field pulls back to L = O/OZ i(OP/9Z-)a3/t. Equip C x JR with Lebesgue measure and consider the orthogonal projection from L2 onto its intersection with the kernel of L. The intersection of this kernel with the Schwartz space may be shown to have infinite dimension, and it is well known that L fails to be C? hypoelliptic. However C? hypoellipticity does hold in a modified form (see [30]): If Lu E C? in an open set, and if in addition u = Lv for some v E L2, where L is the formal adjoint of L, then u E C? there. Our main purpose is to point out that the analogous assertion for real-analytic hypoellipticity is false if P(z) = [Rz]m, m an even integer > 4. Specifically, for P a general, homogeneous, subharmonic, nonharmonic polynomial, let S((z, t); (w, s)) be the Szeg6 kernel; that is, the distribution kernel associated to the operator defined by the orthogonal projection of L2(C x JR), with respect to Lebesgue measure, onto the kernel of L. We shall see in Section 5 that this projection maps Co to C?. Further, S is smooth off the diagonal by [37]. Define the distribution