Motion Segmentation from Optical Flow

Abstract

This note describes two different methods for motion segmentation from optical flow. In the first method, the Jacobian matrix of the first spatial derivatives of the components of optical flow is used to compute the amount of uniform expansion, pure rotation, and shear at every flow point. In the second description, local properties of optical flow which are invariant for non-singular linear transformation are computed from the trace and the determinant of the matrix itself. Both the methods allow to distinguish between different kinds of motion, like translation, rotation, and relative motion in sequences of time-varying images. Preliminary results show that they can also be useful to identify the different moving objects in the viewed scene. Time-varying images provide useful information for the understanding of several visual problems. This information, which can be thought of as encoded in optical flow [1] — a dense planar vector field which gives the velocity of points over the image plane —, appears to be essential for important visual tasks like passive navigation and dynamic scene understanding (see [2-5] for example). In this note, the optical flow computed through a technique which has recently been proposed in ref. [6,7] is used for motion segmentation. Two different local descriptions of image motion that are invariant for orthogonal transformations of coordinates on the image plane (i.e., arbitrary rotation of the viewing camera around the optical axis) are discussed. In the first description, which is obtained by looking at the changing image brightness as a ID deformable body, the Jacobian matrix of the first spatial derivatives of the components of optical flow is used to compute the amount of uniform expansion, pure rotation, and shear at every flow point. The second description focuses on the local properties of optical flow which are invariant for non-singular linear transformation and thus can be inferred from the trace and the determinant of the matrix itself. According to these descriptions, image motion can be segmented in regions where either the average percentage of uniform expansion, pure rotation, or shear is larger than a fixed value, or where the qualitative nature of the eigenvalues do not change. Experiments on real images show that, in many cases, these regions roughly correspond to the image of the observed moving objects and make it possible to distinguish between different kinds of 3D motions. The computation of the percentage of uniform expansion, pure rotation, and shear seems to be less sensitive to noise than the study of the qualitative nature of the eigenvalues of the Jacobian matrix. In the case of translation, which is discussed analytically, the percentage of uniform expansion is usually much larger than the percentages of pure rotation and shear. At the same time, the two eigenvalues of the Jacobian matrix are real and often almost equal. At boundary points, instead, the shear component is larger and the eigenvalues may have opposite sign. Motion segmentation for rotation and relative motion is also discussed. Finally, it is shown that the integration between the presented motion segmentation and other visual cues like intensity edges allows to obtain more accurate image segmentation. Some conclusions can be drawn from the presented analysis. Firstly, it has been shown that it is possible to obtain motion segmentation from optical flow. Two different techniques based on the study of the Jacobian matrix of optical flow have been implemented which can be used to segment the image plane in regions that allow to distinguish between different kinds of motion, like translation, rotation, and relative motion, and to identify the different moving objects. The presented results complement recent results [8] that have been obtained on qualitative and quantitative properties of the Jacobian matrix at the singular points of optical flow, that is, the points where the flow vanishes. Here, the segmentation and the analysis of the spatial structure of optical flow in the neighborhood of singular points, which were essential for the understanding of the observed 3D motion in ref. [8], are obtained easily and reliably from local analysis. In fact, the obtained motion segmentation is useful even when no singular point is found in optical flow. Finally, it appears that the technique which has been used for the computation of optical flow [6,7] is not only adequate for 3D motion recovery from singular points of optical flow [6,7,9], but also for motion and object segmentation.