Context Enhancement for Linear Contextual Multi-Armed Bandits

Abstract

Contextual Multi-Armed Bandit (CMAB) algorithms such as LinUCB (Linear Upper Confidence Bound) or Contextual Thompson Sampling (CTS) base their resolution on the assumption that there exists a linear dependency between the expected reward of an action and its context. Since context constrains such sequential decision problems, it seems unavoidable to work, as far as possible, with the most relevant context features. This article first sheds light on some contextual issues that can be encountered by real-world applications, and then proposes a new method of context enhancement, called Individual Context Enrichment: ICE, to be combined with CMABs. ICE allows CMAB algorithms to rely on additional relevant context features that are computed according to the previously obtained individual accuracy of users when in given contexts which we define as user-context pairs. Basically, ICE classifies user-context pairs according to their individual accuracy, and uses the obtained classes to enrich the original context. To be effective, our method requires regular users, thus it is particularly interesting in the case of applications having identifiable subscribers e.g., recommender systems, clinical trial or mobile health. We experiment and discuss our method, which shows better results on several real-life datasets in terms of global accuracy and cumulative regrets than any other original competitive Multi-Armed Bandit (MAB) or CMAB methods.