Evolving multi-user fuzzy classifier systems integrating human uncertainty and expert knowledge

Abstract

Evolving classifiers and especially evolving fuzzy classifiers have been established as a prominent technique for addressing the recent demands in building classifiers in an incremental open-loop manner, e.g. for the purpose of processing data streams online. So far, the focus lies on classifiers which are obtained based on the input and/or feedback in the form of target labels provided by a single user/expert. In this paper, we propose three variants of evolving multi-user fuzzy classifier systems (EFCS-MU), where multiple users may provide their label feedback: i) ensembled single-user classifiers system, which allows a separate classifier training per user and embeds an advanced aggregation strategy (→ ensembling on a model level), ii) consensus all-user classifier system, where a joint classifier is established for all users based on consensus labelings (→ ensembling on a label level), iii) shift-work all-user classifier system, where a joint classifier is established for all users based on the classical shift-work concept. The classifiers are incrementally evolved by a single-pass learning approach embedding the autonomous evolution of new rules on demand; it integrates an unsupervised evolving clustering technique for rule partitioning, thus the same partition is established in all single-user classifiers, only the consequents in the form of class confidence vectors typically differ among the users due to their different labelings. This offers direct explainability of the varying users’ annotation behaviors. The possible different experience levels of the users in relation to the process behind and possible ambiguities among the provided users’ labels are handled by the proper integration of uncertainty levels into the update of the classifier(s). Furthermore, a concept is presented as to how to adequately integrate possibly available expert rules for a particular newly (on-the-fly) arising class (or in advance for several classes). Finally, an on-line active learning (oAL) strategy is demonstrated, to select only the most important samples to be labelled and thus reduce users’ labeling costs, ensuring economically practicable applicability. The approach was successfully evaluated on two real-world application scenarios, one stemming for a visual inspection scenario, where four users check the quality of the imprint of compact discs and are affected by different experience levels, and one from a nursery school employment ranking application, where a new class was introduced later. The results provide insights into the performance behavior of the three different multi-user classifier variants under different circumstances (with and without expert rules, uncertainty integration, different labelling budgets etc.), including comparisons based on on-line accuracy trends versus the economy of the labeling effort.