Evaluating machine learning technologies for food computing from a data set perspective

Abstract

Food plays an important role in our lives that goes beyond mere sustenance. Food affects behavior, mood, and social life. It has recently become an important focus of multimedia and social media applications. The rapid increase of available image data and the fast evolution of artificial intelligence, paired with a raised awareness of people’s nutritional habits, have recently led to an emerging field attracting significant attention, called food computing, aimed at performing automatic food analysis. Food computing benefits from technologies based on modern machine learning techniques, including deep learning, deep convolutional neural networks, and transfer learning. These technologies are broadly used to address emerging problems and challenges in food-related topics, such as food recognition, classification, detection, estimation of calories and food quality, dietary assessment, food recommendation, etc. However, the specific characteristics of food image data, like visual heterogeneity, make the food classification task particularly challenging. To give an overview of the state of the art in the field, we surveyed the most recent machine learning and deep learning technologies used for food classification with a particular focus on data aspects. We collected and reviewed more than 100 papers related to the usage of machine learning and deep learning for food computing tasks. We analyze their performance on publicly available state-of-art food data sets and their potential for usage in multimedia food-related applications for various needs (communication, leisure, tourism, blogging, reverse engineering, etc.). In this paper, we perform an extensive review and categorization of available data sets: to this end, we developed and released an open web resource in which the most recent existing food data sets are collected and mapped to the corresponding geographical regions. Although artificial intelligence methods can be considered mature enough to be used in basic food classification tasks, our analysis of the state-of-the-art reveals that challenges related to the application of this technology need to be addressed. These challenges include, among others: poor representation of regional gastronomy, incorporation of adaptive learning schemes, and reverse engineering for automatic food creation and replication.