Empirical Evaluation of Computer Vision Algorithms

Abstract

——————————F—————————— omputer vision emerged as a subfield in computer science and in electrical engineering in the 1960s. Two main motivations for research in computer vision are to develop algorithms to solve vision problems and to understand and model the human visual system. It turns out that finding satisfactory answers to either motivation is significantly harder than common wisdom initially assumed. Research in computer vision has actively continued to the current time. Most of the research in the computer vision and pattern recognition community is focused on developing solutions to vision problems. With three decades of research behind current efforts and with the availability of powerful, inexpensive computers, there is a common belief that computer vision is poised to deliver reliable solutions. The area of empirical evaluation of computer vision algorithms is developing the methods and tools for measuring the ability of algorithms to meet requirements to be fielded, for determining the state-of-the-art, and for pointing out future research directions. The goal of this special theme section of IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI) is to highlight progress in empirical evaluation and identify it as a maturing area of computer vision. Out of 18 submissions, three were accepted for this special section. In addition, one submission was accepted to appear in a regular issue, and two others are being revised for consideration as regular papers. “Filtering for Supervised Texture Segmentation: A Comparative Study” by T. Randen and J.H. Husoy presents a comparative study of methods for texture classification. The emphasis of the study is filtering methods from signal processing. Most major filtering approaches are evaluated. For reference, a statistical algorithm and a model-based algorithm are also evaluated. The paper presents performance results on a number of mosaic texture images. In a first for PAMI, the raw image files for these images are being made available as part of the electronic version of the paper. (The electronic version of the paper is part of the Computer Society’s digital library, accessible online at www.computer.org.) It is hoped that future papers on texture segmentation will take advantage of this in order to present directly comparable experimental results. “Performance Evaluation and Analysis of Monocular Building Extraction From Aerial Imagery” by J.A. Shufelt evaluates end-to-end performance of four systems on their ability to extract buildings from 83 aerial images of 18 sites. The methodology allows for an examination of traditional assumptions made in designing algorithms that extract buildings from monocular imagery. “Evaluation of Methods for Ridge and Valley Detection” by A.M. Lopez, F. Lumbreras, and J. Serrat evaluates ridge and valley detectors. The authors discuss what are desirable properties of ridge and valley detectors and the methods for measuring desirable properties. Then they present an evaluation using these methods. We hope the papers in this special section are interesting and present challenges for future researchers.