Concept Drift Handling in Information Systems: Preserving the Validity of Deployed Machine Learning Models

Abstract

Predictions computed by supervised machine learning models play a crucial role in a variety of innovative applications in business and industry. Typically, value is generated as soon as these models are deployed and continuously used in information systems of an organization. However, machine learning endeavors predominantly focus on conceiving applications for static situations. In this context, the management of the models’ lifecycle to preserve their effectiveness over time in dynamic environments is still in its infancy. Therefore, this thesis starts with systematically analyzing the full lifecycle of machine learning applications from an information systems (IS) perspective—and understanding and documenting all choices that have to be made throughout this cycle. On that basis, we then perform a qualitative study via practitioner interviews to map particular challenges in the deployment phase. In this context, we identify concept drift as a particularly important challenge to overcome: Concept drift refers to changes in the environment over time which affect the behavior of a machine learning model. This can have an impact on the model’s prediction quality and its overall utility. We analyze and categorize concept drift handling approaches covering both the detection of concept drift as well as the appropriate adaptation of the model. We identify two particular research gaps: the handling of concept drift for regression tasks and the handling of concept drift for tasks where additional labels for retraining a model are hard or costly to obtain. For both areas, we develop new methods and demonstrate their effectiveness in technical experiments and real-world use cases. This thesis contributes new methods to handle concept drift, for particular difficult contexts: First, this thesis should raise researchers’ and practitioners’ awareness for the topic of changing input data over time as well as for its impact on deployed machine learning. Second, the developed and tested methods can either be implemented directly or serve as inspiration to conceive appropriate drift handling strategies within information systems. Finally, we expect that advanced concept drift handling not only technically ensures a reliable prediction quality over time, but that it will also increase trust and acceptance of machine learning-based information systems—and, thus, help to boost the impact of machine learning.