User-driven sampling strategies in image exploitation

Abstract

Both visual analytics and interactive machine learning try to leverage the complementary strengths of humans and machines to solve complex data exploitation tasks. These fields overlap most significantly when training is involved: the visualization or machine learning tool improves over time by exploiting observations of the human–computer interaction. This article focuses on one aspect of the human–computer interaction that we call user-driven sampling strategies. Unlike relevance feedback and active learning sampling strategies, where the computer selects which data to label at each iteration, we investigate situations where the user selects which data are to be labeled at each iteration. User-driven sampling strategies can emerge in many visual analytics applications, but they have not been fully developed in machine learning. User-driven sampling strategies suggest new theoretical and practical research questions for both visualization science and machine learning. In this article, we identify and quantify the potential benefits of these strategies in a practical image analysis application. We find user-driven sampling strategies can sometimes provide significant performance gains by steering tools toward local minima that have lower error than tools trained with all of the data. In preliminary experiments, we find these performance gains are particularly pronounced when the user is experienced with the tool and application domain.