Using Individual Accuracy to Create Context for Non-Contextual Multi-Armed Bandit Problems

Abstract

Contextual Multi-Armed Bandit (CMAB) algorithms suppose a linear dependency between the expected reward of an action and its context. If given a complete and relevant context they eventually provide full personalization. However, when context is sparse or missing entirely they fail to solve the problem efficiently. In this article, we give an overview of the Individual Context Enrichment (ICE) method, initially designed to enrich sparse context. Our main contribution is the use of the ICE method with CMAB problems when no context information is available. We propose to measure the levels of performance of a MAB algorithm for its different users and utilize it as context information. This way we transform a hidden context Multi-Armed Bandit problem into an equivalent contextual problem that can be worked on by CMAB algorithms. To be effective, our method requires regular and identifiable users, thus it is particularly interesting in the case of applications having subscribers e.g., recommender systems, clinical trials or mobile health. Our method has been experimented on several datasets and shows better results in terms of accuracy and cumulative regrets than Thompson Sampling, a competitive MAB method.