Tracking millions of query intents with StarrySky and its applications

Abstract

Query intent mining is a critical problem in various real-world search applications. In the past few years, dramatic advances have been witnessed in the field of query intent mining. This article presents a practical system – StarrySky for identifying and inferring millions of fine-grained query in tents in daily sponsored search with high precision, and shows how real-world applications can benefit from query intent tracking results using StarrySky. Great advantages have already been achieved by deploying this system in Sogou sponsored search engine. The general architecture of StarrySky consists of three stages. First, millions of fine-grained query clusters are detected from two years’ web search click logs by using a well-defined community detection algorithm. Second, query intents named as concepts are found by refining the query clusters with a series of well-designed operations. Third and foremost, a real-time inference algorithm is built for precisely assigning query intents to the detected concepts. Aside from the description of the system, several experiments are conducted to evaluate its performance. The inference algorithm achieves up to 96% precision and 68% coverage on daily search requests. It is also demonstrated that StarrySky is a practical and valuable system for tracking query intents in several real-world applications.