Abstract
Unsupervised machine learning is the training of an artificial intelligence system using information that is neither classified nor labeled, with a view to modeling the underlying structure or distribution in a dataset. Since unsupervised machine learning systems are widely used in many real-world applications, assessing the appropriateness of these systems and validating their implementations with respect to individual users' requirements and specific application scenarios/contexts are indisputably two important tasks. Such assessments and validation tasks, however, are fairly challenging due to the absence of a priori knowledge of the data. In view of this challenge, in this article, we develop a METamorphic Testing approach to assessing and validating unsupervised machine LEarning systems, abbreviated as mettle. Our approach provides a new way to unveil the (possibly latent) characteristics of various machine learning systems, by explicitly considering the specific expectations and requirements of these systems from individual users' perspectives. To support mettle, we have further formulated 11 generic metamorphic relations (MRs), covering users' generally expected characteristics that should be possessed by machine learning systems. We have performed an experiment and a user evaluation study to evaluate the viability and effectiveness of mettle. Our experiment and user evaluation study have shown that, guided by user-defined MR-based adequacy criteria, end users are able to assess, validate, and select appropriate clustering systems in accordance with their own specific needs. Our investigation has also yielded insightful understanding and interpretation of the behavior of the machine learning systems from an end-user software engineering's perspective, rather than a designer's or implementor's perspective, who normally adopts a theoretical approach.
Highlights
U NSUPERVISED machine learning requires no prior knowledge and can be widely used in a large variety of applications such as market segmentation for targeting customers [1], anomaly or fraud detection in banking [2], grouping genes or proteins in biological process [3], deriving climate indices from earth science data [4], and document clustering based on content [5]
1) We proposed an metamorphic testing (MT)-based approach (METTLE) to assessing and validating unsupervised machine learning systems that generally suffer from the absence of a priori knowledge of the data and a test oracle
As for MR5.1, we found that the violations were mainly due to the data normalization task during the preprocessing stage, and the effects of normalization varied across different violations
Summary
U NSUPERVISED machine learning requires no prior knowledge and can be widely used in a large variety of applications such as market segmentation for targeting customers [1], anomaly or fraud detection in banking [2], grouping genes or proteins in biological process [3], deriving climate indices from earth science data [4], and document clustering based on content [5]. (In this article, end users, or users, refer to those people who are “causal” users of clustering systems They have some hands-on experience on using such systems, they often do not possess a solid theoretical foundation on machine learning. These users come from different fields such as bioinformatics [10] and nuclear engineering [11] Their main concern is the applicability of a clustering system in the users’ specific contexts, rather than the detailed logic of this system.) From a user’s perspective, this selection is not trivial [12], because end users generally do not have very solid theoretical background on machine learning, and because the selection task involves two complex issues as follows.
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
