Two convex counterexamples: A discontinuous envelope function and a nondifferentiable nearest-point mapping

Abstract

Introduction. This paper gives counterexamples for two unrelated conjectures which pertain to convexity theory. Supposef is any function defined on a closed convex set C in finitedimensional Euclidean space. Its envelope function env f is defined as the pointwise supremum of all linear functions which f dominates everywhere. It is elementary and well known that env f is convex, and hence continuous in the relative interior of C. It is similarly known that env f is lower-semicontinuous on the boundary of C. Witsenhausen [14] uses and studies envelope functions. He suggested the following orally: CONJECTURE 1. Iff is continuous everywhere on C, then envf is also continuous everywhere on C. I provide a very simple counterexample in 3-space which shows that this is not true. Incidentally, though the proof is not on paper, I feel fairly certain that Witsenhausen's conjecture is correct in 2-space. If C is a closed convex set in finite-dimensional Euclidean space, then for any point x in the space there is a unique point P(x) in C which is nearest to x. The mapping P is called the projection onto C. (Many interesting results about this mapping are given in Phelps [9], [10].) It is easy to prove and well known that P is a contraction mapping, hence continuous. It is also easy to see that P need not have much in the way of differentiability properties, for if C is a polyhedron in 2-space, then it is easy to find points and directions at which P does not even have a two-sided directional derivative. The existence in general of a one-sided directional derivative, however, is not so easy to decide. This question does not seem to be raised in such natural places as Eggleston [4] and Valentine [11]. CONJECTURE 2. For all points and directions, P has a one-sided directional derivative. My mildly complicated counterexample is a set C (in 3-space), which is the convex hull of a countable infinity of points. I suspect that the conjecture is correct in 2-space. In n-space it would be interesting to know whether the set of points and directions at which the conjecture holds is dense in EnXSn-. If not, the convex set would have to be extremely jagged!