Using Machine Learning to Improve Lead Times in the Identification of Emerging Customer Needs

Abstract

In recent years, computational approaches for automatically extracting the voice of the customer from user generated content have been proposed. These studies have tackled the task of obtaining current customer needs, however, there is a lack of methods that predict future needs (i.e. needs that may become popular in the marketplace). Therefore, this study presents a multi-document keyphrase extraction algorithm which predicts future customer needs from users’ social media posts on Reddit. Key to our approach is a novel document filtering method (discovering potentially relevant social media content) and a keyphrase ranking method, which promotes terms with rising frequency likely to be future product needs. In order to evaluate the approach, a case study of “toothpaste” needs is reviewed and a novel evaluation approach using ground truth automatically extracted from a collection of future specifications of new-to-market products is proposed. In our evaluation, we show that the approach is significantly better than simple baselines at identifying customer needs on social media before they trend in the marketplace. We also show that our approach can capture important customer needs identified by a large multinational company with lead times of up to 25 months ahead of them trending in the marketplace. The findings of this research could provide many benefits to businesses such as gaining early access into markets ahead of their competitors and giving early notice to manufacturers/engineers/developers before a need for a product is in demand.