How and What Can Humans Learn from Being in the Loop?

Nicolas Pfeuffer,Oliver Hinz,Benjamin M Abdel-Karim,Gernot Rohde

doi:10.1007/s13218-020-00638-x

Abstract

This article discusses the counterpart of interactive machine learning, i.e., human learning while being in the loop in a human-machine collaboration. For such cases we propose the use of a Contradiction Matrix to assess the overlap and the contradictions of human and machine predictions. We show in a small-scaled user study with experts in the area of pneumology (1) that machine-learning based systems can classify X-rays with respect to diseases with a meaningful accuracy, (2) humans partly use contradictions to reconsider their initial diagnosis, and (3) that this leads to a higher overlap between human and machine diagnoses at the end of the collaboration situation. We argue that disclosure of information on diagnosis uncertainty can be beneficial to make the human expert reconsider her or his initial assessment which may ultimately result in a deliberate agreement. In the light of the observations from our project, it becomes apparent that collaborative learning in such a human-in-the-loop scenario could lead to mutual benefits for both human learning and interactive machine learning. Bearing the differences in reasoning and learning processes of humans and intelligent systems in mind, we argue that interdisciplinary research teams have the best chances at tackling this undertaking and generating valuable insights.

Highlights

Machine learning (ML) and especially deep learning (DL) have seen a dramatic resurgence in the past decade, largely driven by increases in computational power and the availability of large datasets that enable a faster training and more accurate models [38]
The data we gathered from our analyses and observations are sufficient to take the first steps in illuminating, whether humans are able to learn from ML-based decision support systems (DSS)
Whenever systems and humans assessing the same material, we propose to present a matrix that we call a “Contradiction Matrix” which helps to understand the learning effect that might occur in the particular setting

Summary

Introduction

Machine learning (ML) and especially deep learning (DL) have seen a dramatic resurgence in the past decade, largely driven by increases in computational power and the availability of large datasets that enable a faster training and more accurate models [38]. The victory of the AlphaGo algorithm over the world champion of the highly. AlphaGo Zero’s winning strategies were previously unknown and difficult to identify, understand and combat for humans and for AlphaGo. AlphaGo Zero’s winning strategies were previously unknown and difficult to identify, understand and combat for humans and for AlphaGo This victory of AlphaGo and the advancements of AlphaGo Zero were regarded as major milestones in artificial intelligence (AI) research itself [28], it could constitute the advent of an important new pattern for human-machine collaboration. Fan Hui, former European champion of Go who had lost to AlphaGo, mentioned the following in an interview with nature news: “[...] The problem is humans

Objectives

Methods

Results

Discussion

Conclusion