Customer behavior classification using surveillance camera for marketing

Abstract

The analysis of customer behavior from surveillance camera is one of the most important open topics for marketing. Traditionally, retailers use the records of cash registers or credit cards to analyze the buying behaviors of customers. However, this information cannot reveal the behaviors of customer when he or she shows interest on the front of the merchandise shelf but does not buy. Those behaviors can be recorded and analyzed by the surveillance camera. We propose a system to classify different customer behaviors on the front of shelf: no interest, viewing, turning body to shelf, touching, picking and returning to shelf and picking and putting into basket, which show customer's increasing interest to products. In the proposed system, head orientation, body orientation, and arm action, the multiple cues are integrated for the customer behavior recognition. The proposed system discretizes the head and body orientation of customer into 8 directions to estimate whether the customer is looking or turning to the merchandise shelf. Semi-Supervised Learning method is applied to optimize the training dataset and to generate the accurate classifier. In addition, the temporal constraint and the human physical model constraint are considered in joint body and head orientation estimation. As for the arm action recognition, a novel Combined Hand Feature (CHF), which includes hand trajectory, tracking status and the relative position between hand and shopping basket, is proposed to classify different arm actions. The hand tracking is done by an improved particle filter. The CHF is classified by Dynamic Bayesian Network (DBN) to output different types of arm actions. A series of experiments demonstrate effectiveness of the proposed technologies and the performance to the developed system.