Shells of matrices in indefinite inner product spaces

Abstract

The notion of the shell of aHilbert spa ce opera tor, which is auseful genera liza tion (proposed by Wielandt) of the numerical range, is extended to operators in spaces with an indefinite inner product. For the most part, finite dimensional spaces are considered. Geometric properties of shells (convexity, boundedness, being a subset of a line, etc.) are described, as well as shells of operators in two dimensional indefinite inner product spaces. For normal operators, it is conjectured that the shell is convex and its closure is polyhedral; the conjecture is proved for indefinite inner product spaces of dimension at most three, and for finite dimensional inner product spaces with one positive eigenvalue.