Product verification using OCR classification and Mondrian conformal prediction

Abstract

The retail sector is undergoing an apparent digital transformation that completely revolutionises shopping operations. To stay competitive, retailer stakeholders are forced to rethink and improve their business models to provide an attractive personalised experience to consumers. The self-service checkout process is at the heart of this transformation and should be designed to identify the products accurately and detect any possible anomalous behaviour. In this paper, we introduce a product verification system based on OCR classification and Mondrian conformal prediction. The proposed system includes three components: OCR reading, text classification and product verification. By using image data from existing grocery stores, the system can detect anomalies with high performance, even when there is partial text information on the products. This makes the system applicable for reducing shrinkage loss (caused, for example, by employee theft or shoplifting) in grocery stores by identifying fraudulent behaviours such as barcode switching and miss-scan. Additionally, OCR reading with NLP classification shows that it is in itself a powerful classifier of products.