Fundamental solutions and critical Lane-Emden exponents for nonlinear integral operators in cones

Abstract

In this article we study the fundamental solutions or “α-harmonic functions” for some nonlinear positive homogeneous nonlocal elliptic problems in conical domains, such asF(u)=0inCω,u=0inRN∖Cω, where ω is a proper C2 domain in SN−1 for N≥2, Cω:={x:x≠0,|x|−1x∈ω} is the cone-like domain related to ω, and F is an extremal fully nonlinear integral operator. We prove the existence of two fundamental solutions that are homogeneous and do not change signs in the cone; one is bounded at the origin and the other at infinity.As an application, we use the fundamental solutions obtained to prove Liouville type theorems in cones for supersolutions of the Lane-Emden-Fowler equation in the formF(u)+up=0inCω,u=0inRN∖Cω. We also prove a generalized Hopf type lemma in domains with corners. Most of our results are new even when F is the fractional Laplacian operator.