Food Recognition System Research Articles

Explainable deep learning ensemble for food image analysis on edge devices

Food recognition systems recently garnered much research attention in the relevant field due to their ability to obtain objective measurements for dietary intake. This feature contributes to the management of various chronic conditions. Challenges such as inter and intraclass variations alongside the practical applications of smart glasses, wearable cameras, and mobile devices require resource-efficient food recognition models with high classification performance. Furthermore, explainable AI is also crucial in health-related domains as it characterizes model performance, enhancing its transparency and objectivity. Our proposed architecture attempts to address these challenges by drawing on the strengths of the transfer learning technique upon initializing MobiletNetV3 with weights from a pre-trained model of ImageNet. The MobileNetV3 achieves superior performance using the squeeze and excitation strategy, providing unequal weight to different input channels and contrasting equal weights in other variants. Despite being fast and efficient, there is a high possibility for it to be stuck in the local optima like other deep neural networks, reducing the desired classification performance of the model. Thus, we overcome this issue by applying the snapshot ensemble approach as it enables the M model in a single training process without any increase in the required training time. As a result, each snapshot in the ensemble visits different local minima before converging to the final solution which enhances recognition performance. On overcoming the challenge of explainability, we argue that explanations cannot be monolithic, since each stakeholder perceive the results’, explanations based on different objectives and aims. Thus, we proposed a user-centered explainable artificial intelligence (AI) framework to increase the trust of the involved parties by inferencing and rationalizing the results according to needs and user profile. Our framework is comprehensive in terms of a dietary assessment app as it detects Food/Non-Food, food categories, and ingredients. Experimental results on the standard food benchmarks and newly contributed Malaysian food dataset for ingredient detection demonstrated superior performance on an integrated set of measures over other methodologies.

Smartphone-based food recognition system using multiple deep CNN models

People with blindness or low vision utilize mobile assistive tools for various applications such as object recognition, text recognition, etc. Most of the available applications are focused on recognizing generic objects. And they have not addressed the recognition of food dishes and fruit varieties. In this paper, we propose a smartphone-based system for recognizing the food dishes as well as fruits for children with visual impairments. The Smartphone application utilizes a trained deep CNN model for recognizing the food item from the real-time images. Furthermore, we develop a new deep convolutional neural network (CNN) model for food recognition using the fusion of two CNN architectures. The new deep CNN model is developed using the ensemble learning approach. The deep CNN food recognition model is trained on a customized food recognition dataset.The customized food recognition dataset consists of 29 varieties of food dishes and fruits. Moreover, we analyze the performance of multiple state of art deep CNN models for food recognition using the transfer learning approach. The ensemble model performed better than state of art CNN models and achieved a food recognition accuracy of 95.55 % in the customized food dataset. In addition to that, the proposed deep CNN model is evaluated in two publicly available food datasets to display its efficacy for food recognition tasks.

Study for Food Recognition System Using Deep Learning

Accurate dietary appraisal has been found by literature to be very significant in the evaluation of weight loss treatments. Most current methods of dietary evaluation, however, depend on recollection. The development of a modern computer-based food recognition system for reliable food evaluation is now possible across comprehensive mobile devices as well as rich Cloud services. Fixing the problem of food detection and identification in photos of different kinds of foods. Given the variety of food products with low inter-and high intra-class variations and the limited information in a single picture, the problem is complicated. By propose the overall application of multiple fusion-trained classifiers to achieve increased identification and recognition capabilities on characteristics obtained from various deep models. This paper studied various techniques of food recognition using different approaches and based on several variables, compared their effectiveness. Our study results demonstrate that deep learning overcomes other strategies like manual feature extractors, standard ML algorithms, as well as DL as a practical tool for food hygiene and safety inspections.

The Implementation of an Ingredient-Based Food recognition System using Deep learning

The Implementation of an Ingredient-Based Food recognition System using Deep learning

Neural Architecture Search for Lightweight Neural Network in Food Recognition

Healthy eating is an essential element to prevent obesity that will lead to chronic diseases. Despite numerous efforts to promote the awareness of healthy food consumption, the obesity rate has been increased in the past few years. An automated food recognition system is needed to serve as a fundamental source of information for promoting a balanced diet and assisting users to understand their meal consumption. In this paper, we propose a novel Lightweight Neural Architecture Search (LNAS) model to self-generate a thin Convolutional Neural Network (CNN) that can be executed on mobile devices with limited processing power. LNAS has a sophisticated search space and modern search strategy to design a child model with reinforcement learning. Extensive experiments have been conducted to evaluate the model generated by LNAS, namely LNAS-NET. The experimental result shows that the proposed LNAS-NET outperformed the state-of-the-art lightweight models in terms of training speed and accuracy metric. Those experiments indicate the effectiveness of LNAS without sacrificing the model performance. It provides a good direction to move toward the era of AutoML and mobile-friendly neural model design.

A deep learning based wearable system for food and drink intake recognition

Eating difficulties and the subsequent need for eating assistance are a prevalent issue within the elderly population. Besides, a poor diet is considered a confounding factor for developing chronic diseases and functional limitations. Driven by the above issues, this paper proposes a wrist-worn tri-axial accelerometer based food and drink intake recognition system. First, an adaptive segmentation technique is employed to identify potential eating and drinking gestures from the continuous accelerometer readings. A posteriori, a study upon the use of Convolutional Neural Networks for the recognition of eating and drinking gestures is carried out. This includes the employment of three time series to image encoding frameworks, namely the signal spectrogram, the Markov Transition Field and the Gramian Angular Field, as well as the development of various multi-input multi-domain networks. The recognition of the gestures is then tackled as a 3-class classification problem (‘Eat’, ‘Drink’ and ‘Null’), where the ‘Null’ class is composed of all the irrelevant gestures included in the post-segmentation gesture set. An average per-class classification accuracy of 97.10% was achieved by the proposed system. When compared to similar work, such accurate classification performance signifies a great contribution to the field of assisted living.

Automatic food recognition system for middle‐eastern cuisines

The concerns for a healthier diet are increasing day by day, especially in diabetics wherein the aim of healthier diet can only be achieved by keeping a track of daily food intake and glucose-level. As a consequence, there is an ever-increasing need for automatic tools able to help diabetics to manage their diet and also help physicians to better analyse the effects of various types of food on the glucose-level of diabetics. In this paper, we propose an intelligent food recognition and tracking system for diabetics, which is potentially an essential part of a mobile application that we propose to couple food intake with the blood glucose-level using glucose measuring sensors. For food recognition, we rely on several feature extraction and classification techniques individually and jointly using an early and three different late fusion techniques, namely (i) Particle Swarm Optimisation (PSO), (ii) Genetic Algorithms (GA) based fusion and (iii) simple averaging. Moreover, we also evaluate the performance of several handcrafted and deep features and compare the results against state-of-the-art. In addition, we collect a large-scale dataset containing images from several types of local Middle-Eastern food, which is intended to become a powerful support tool for future research in the domain.

DeepFood: Food Image Analysis and Dietary Assessment via Deep Model

Food is essential for human life and has been the concern of many healthcare conventions. Nowadays new dietary assessment and nutrition analysis tools enable more opportunities to help people understand their daily eating habits, exploring nutrition patterns and maintain a healthy diet. In this paper, we develop a deep model based food recognition and dietary assessment system to study and analyze food items from daily meal images (e.g., captured by smartphone). Specifically, we propose a three-step algorithm to recognize multi-item (food) images by detecting candidate regions and using deep convolutional neural network (CNN) for object classification. The system first generates multiple region of proposals on input images by applying the Region Proposal Network (RPN) derived from Faster R-CNN model. It then indentifies each region of proposals by mapping them into feature maps, and classifies them into different food categories, as well as locating them in the original images. Finally, the system will analyze the nutritional ingredients based on the recognition results and generate a dietary assessment report by calculating the amount of calories, fat, carbohydrate and protein. In the evaluation, we conduct extensive experiments using two popular food image datasets - UEC-FOOD100 and UEC-FOOD256. We also generate a new type of dataset about food items based on FOOD101 with bounding. The model is evaluated through different evaluation metrics. The experimental results show that our system is able to recognize the food items accurately and generate the dietary assessment report efficiently, which will benefit the users with a clear insight of healthy dietary and guide their daily recipe to improve body health and wellness.

A New Deep Learning-Based Food Recognition System for Dietary Assessment on An Edge Computing Service Infrastructure

Literature has indicated that accurate dietary assessment is very important for assessing the effectiveness of weight loss interventions. However, most of the existing dietary assessment methods rely on memory. With the help of pervasive mobile devices and rich cloud services, it is now possible to develop new computer-aided food recognition system for accurate dietary assessment. However, enabling this future Internet of Things-based dietary assessment imposes several fundamental challenges on algorithm development and system design. In this paper, we set to address these issues from the following two aspects: (1) to develop novel deep learning-based visual food recognition algorithms to achieve the best-in-class recognition accuracy; (2) to design a food recognition system employing edge computing-based service computing paradigm to overcome some inherent problems of traditional mobile cloud computing paradigm, such as unacceptable system latency and low battery life of mobile devices. We have conducted extensive experiments with real-world data. Our results have shown that the proposed system achieved three objectives: (1) outperforming existing work in terms of food recognition accuracy; (2) reducing response time that is equivalent to the minimum of the existing approaches; and (3) lowering energy consumption which is close to the minimum of the state-of-the-art.

An optimized convolutional neural network with bottleneck and spatial pyramid pooling layers for classification of foods

Keeping record of daily meal intake is an effective solution for tackling with obesity and overweight. This can be done by developing apps on smartphones that are able to automatically recommend a short list of most probable foods by analyzing the photo taken from food. Then, the user chooses the correct answer from the short list. Hence, the automatic food recognition system must be able to recommend an accurate list. In other words, it is not essential for these apps to have a very high top-1 accuracy. Considering that the app will show the list of 5 most probable foods, the food recognition system must have a high top-5 accuracy. A food recognition system is usually developed by adapting knowledge of state-of-the-art networks such as GoogleNet and ResNet to the domain of food. However, these networks have high number of parameters. In this paper, we propose a 23-layer architecture which has 99.14% and 96.63% fewer parameter compared with ResNet and GoogleNet. Our experiment on Food101 and UECFood-256 datasets shows that although our network reduces the number of parameters dramatically, it produces more accurate results than GoogleNet and its accuracy is comparable with ResNet.

Mobile Multi-Food Recognition Using Deep Learning

In this article, we propose a mobile food recognition system that uses the picture of the food, taken by the user’s mobile device, to recognize multiple food items in the same meal, such as steak and potatoes on the same plate, to estimate the calorie and nutrition of the meal. To speed up and make the process more accurate, the user is asked to quickly identify the general area of the food by drawing a bounding circle on the food picture by touching the screen. The system then uses image processing and computational intelligence for food item recognition. The advantage of recognizing items, instead of the whole meal, is that the system can be trained with only single item food images. At the training stage, we first use region proposal algorithms to generate candidate regions and extract the convolutional neural network (CNN) features of all regions. Second, we perform region mining to select positive regions for each food category using maximum cover by our proposed submodular optimization method. At the testing stage, we first generate a set of candidate regions. For each region, a classification score is computed based on its extracted CNN features and predicted food names of the selected regions. Since fast response is one of the important parameters for the user who wants to eat the meal, certain heavy computational parts of the application are offloaded to the cloud. Hence, the processes of food recognition and calorie estimation are performed in cloud server. Our experiments, conducted with the FooDD dataset, show an average recall rate of 90.98%, precision rate of 93.05%, and accuracy of 94.11% compared to 50.8% to 88% accuracy of other existing food recognition systems.

A supervised extreme learning committee for food recognition

A food recognition system exploiting a supervised committee of classifiers is proposed.The system automatically selects the optimal features for the task.The structured fusion approach is designed ...

A supervised extreme learning committee for food recognition

Food recognition is an emerging topic in computer vision. The problem is being addressed especially in health-oriented systems where it is used as a support for food diary applications. The goal is to improve current food diaries, where the users have to manually insert their daily food intake, with an automatic recognition of the food type, quantity and consequent calories intake estimation. In addition to the classical recognition challenges, the food recognition problem is characterized by the absence of a rigid structure of the food and by large intra-class variations. To tackle such challenges, a food recognition system based on a committee classification is proposed. The aim is to provide a system capable of automatically choosing the optimal features for food recognition out of the existing plethora of available ones (e.g., color, texture, etc.). Following this idea, each committee member, i.e., an Extreme Learning Machine, is trained to specialize on a single feature type. Then, a Structural Support Vector Machine is exploited to produce the final ranking of possible matches by filtering out the irrelevant features and thus merging only the relevant ones. Experimental results show that the proposed system outperforms state-of-the-art works on four publicly available benchmark datasets.

"Snap-n-Eat": Food Recognition and Nutrition Estimation on a Smartphone.

We present snap-n-eat, a mobile food recognition system. The system can recognize food and estimate the calorific and nutrition content of foods automatically without any user intervention. To identify food items, the user simply snaps a photo of the food plate. The system detects the salient region, crops its image, and subtracts the background accordingly. Hierarchical segmentation is performed to segment the image into regions. We then extract features at different locations and scales and classify these regions into different kinds of foods using a linear support vector machine classifier. In addition, the system determines the portion size which is then used to estimate the calorific and nutrition content of the food present on the plate. Previous approaches have mostly worked with either images captured in a lab setting, or they require additional user input (eg, user crop bounding boxes). Our system achieves automatic food detection and recognition in real-life settings containing cluttered backgrounds. When multiple food items appear in an image, our system can identify them and estimate their portion size simultaneously. We implemented this system as both an Android smartphone application and as a web service. In our experiments, we have achieved above 85% accuracy when detecting 15 different kinds of foods.

Large Scale Learning for Food Image Classification

Since health care on foods is drawing people's attention recently, in this paper we propose a computer vision based food recognition system could be used to estimate food for diabetes patients. This study proposes a methodology for automatic food recognition, based on the Bag of Features (BoF) model. We present an approach to find out the group and location of objects in images. The system computes dense local features using scale invariant features. It performs very fast classification of each pixel in an image. For the design and valuation of the proposed system, a image dataset with nearly 5010 food images was created and organized into 11 classes. This system has achieved the accuracy of 78%.of objects in images. The system computes dense local features using scale invariant features. It performs very fast classification of each pixel in an image. For the design and valuation of the proposed system, a image dataset with nearly 5010 food images was created and organized into 11 classes. This system has achieved the accuracy of 78%. DOI: 10.17762/ijritcc2321-8169.150317

A food recognition system for diabetic patients based on an optimized bag-of-features model.

Computer vision-based food recognition could be used to estimate a meal's carbohydrate content for diabetic patients. This study proposes a methodology for automatic food recognition, based on the bag-of-features (BoF) model. An extensive technical investigation was conducted for the identification and optimization of the best performing components involved in the BoF architecture, as well as the estimation of the corresponding parameters. For the design and evaluation of the prototype system, a visual dataset with nearly 5000 food images was created and organized into 11 classes. The optimized system computes dense local features, using the scale-invariant feature transform on the HSV color space, builds a visual dictionary of 10000 visual words by using the hierarchical k-means clustering and finally classifies the food images with a linear support vector machine classifier. The system achieved classification accuracy of the order of 78%, thus proving the feasibility of the proposed approach in a very challenging image dataset.

FoodCam: A real-time food recognition system on a smartphone

We propose a mobile food recognition system, FoodCam, the purposes of which are estimating calorie and nutrition of foods and recording a user's eating habits. In this paper, we propose image recognition methods which are suitable for mobile devices. The proposed method enables real-time food image recognition on a consumer smartphone. This characteristic is completely different from the existing systems which require to send images to an image recognition server. To recognize food items, a user draws bounding boxes by touching the screen first, and then the system starts food item recognition within the indicated bounding boxes. To recognize them more accurately, we segment each food item region by GrubCut, extract image features and finally classify it into one of the one hundred food categories with a linear SVM. As image features, we adopt two kinds of features: one is the combination of the standard bag-of-features and color histograms with ?2 kernel feature maps, and the other is a HOG patch descriptor and a color patch descriptor with the state-of-the-art Fisher Vector representation. In addition, the system estimates the direction of food regions where the higher SVM output score is expected to be obtained, and it shows the estimated direction in an arrow on the screen in order to ask a user to move a smartphone camera. This recognition process is performed repeatedly and continuously. We implemented this system as a standalone mobile application for Android smartphones so as to use multiple CPU cores effectively for real-time recognition. In the experiments, we have achieved the 79.2 % classification rate for the top 5 category candidates for a 100-category food dataset with the ground-truth bounding boxes when we used HOG and color patches with the Fisher Vector coding as image features. In addition, we obtained positive evaluation by a user study compared to the food recording system without object recognition.