Customer pose estimation using orientational spatio-temporal network from surveillance camera

Abstract

The analysis of customer pose draws more and more attention of retailers and researchers, because this information can reveal the customer habits and the customer interest level to the merchandise. In the retail store environment, customers' poses are highly related to their body orientations. For example, when a customer is picking an item from merchandise shelf, he or she must face to the shelf. On the other hand, if the customer body orientation is parallel to the shelf, this customer is probably just walking through. Considering this fact, we propose a customer pose estimation system using orientational spatio-temporal deep neural network from surveillance camera. This system first generates the initial joint heatmaps using a fully convolutional network. Based on these heatmaps, we propose a set of novel orientational message-passing layers to fine-tune joint heatmaps by introducing the body orientation information into the conventional message-passing layers. In addition, we apply a bi-directional recurrent neural network on top of the system to improve the estimation accuracy by considering both forward and backward image sequences. Therefore, in this system, the global body orientation, local joint connections, and temporal pose continuity are integrally considered. At last, we conduct a series of comparison experiments to show the effectiveness of our system.