Abstract
Eliciting knowledge from domain experts can play an important role throughout the machine learning process, from correctly specifying the task to evaluating model results. However, knowledge elicitation is also fraught with challenges. In this work, we consider why and how machine learning researchers elicit knowledge from experts in the model development process. We develop a taxonomy to characterize elicitation approaches according to the elicitation goal, elicitation target, elicitation process, and use of elicited knowledge. We analyze the elicitation trends observed in 28 papers with this taxonomy and identify opportunities for adding rigor to these elicitation approaches. We suggest future directions for research in elicitation for machine learning by highlighting avenues for further exploration and drawing on what we can learn from elicitation research in other fields.
Highlights
While many research articles on Machine learning (ML)-advised decision making may mention the importance of human knowledge, given potential differences between how domain experts versus novices represent and articulate their knowledge, we focused on research aimed at eliciting expertise that was implied to be held by certain groups of professionals, and could not be obtained from the online or university student recruitment pools
We describe observed differences in elicitation paths based on their goals, and point to gaps and opportunities for knowledge elicitation in ML that emerge from our analysis
Eliciting expert knowledge is a standard part of many machine learning workflows
Summary
Machine learning (ML) technologies are integrated into a diverse swathe of datadriven decision-making applications. Declarative, procedural, and conditional information that a person possesses related to a particular domain—is a common goal. Expert opinion and judgment enter into the practice of statistical inference and decision-making in myriad ways across many domains. By obtaining and using expert knowledge, ML engineers or researchers can produce more robust, accurate, and trustworthy models
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
