How Many Data Does Machine Learning in Human–Computer Interaction Need?: Re-Estimating the Dataset Size for Convolutional Neural Network-Based Models of Visual Perception

Abstract

Artificial intelligence (AI)-based user-interface (UI) design and evaluation are currently constrained by the scarcity of human-generated training data. Correspondingly, choosing appropriate neural network (NN) architecture and carefully planning the sample size is essential for building accurate machine learning models. Previously, we have estimated that for a convolutional NN to produce better mean square errors (MSEs) than feature-based models, the required training dataset size should be approximately 3000. Our current validation study with roughly 4000 web UIs and 233 subjects suggests that the estimation should be closer to 17,000. We propose corrected regression models suggesting that the dataset size effect is better described using a logarithmic function. We also report significant differences in MSEs between the employed perception dimensions, with Aesthetics models having an MSE 21.5% worse than Complexity and 12.1% worse than Orderliness.