Food Image Analysis Research Articles

Food Image Detection Using CNN Algorithm

In the coming years, the fusion of deep learning with computer vision will usher in significant advancements in various domains, including food image analysis. A pioneering approach will be introduced to meet the rapidly increasing demand for automated recipe generation from food images, leveraging state-of-the-art deep learning techniques. The proliferation of social media platforms and the omnipresence of food-related content will necessitate efficient algorithms capable of understanding food images and generating coherent recipes. The proposed framework will integrate deep learning neural networks for robust food image recognition and sequence-to-sequence models for recipe generation. Initially, a pre-trained model will be employed on extensive food image datasets to extract pertinent features, followed by fine-tuning on domain-specific datasets to enhance accuracy. This pre-training step will enable the network to learn rich representations of various food items and their compositions. Subsequently, a sequence-to-sequence model, such as an encoder-decoder architecture equipped with attention mechanisms, will be employed to map the extracted image features to textual recipes. This model will learn to generate coherent and contextually relevant recipes from input food images, capturing the intricate details of different cuisines and dishes. Key Words: Deep Learning, Food Image Analysis, Recipe Generation, Convolutional Neural Networks.

Mid-DeepLabv3+: A Novel Approach for Image Semantic Segmentation Applied to African Food Dietary Assessments.

Recent decades have witnessed the development of vision-based dietary assessment (VBDA) systems. These systems generally consist of three main stages: food image analysis, portion estimation, and nutrient derivation. The effectiveness of the initial step is highly dependent on the use of accurate segmentation and image recognition models and the availability of high-quality training datasets. Food image segmentation still faces various challenges, and most existing research focuses mainly on Asian and Western food images. For this reason, this study is based on food images from sub-Saharan Africa, which pose their own problems, such as inter-class similarity and dishes with mixed-class food. This work focuses on the first stage of VBDAs, where we introduce two notable contributions. Firstly, we propose mid-DeepLabv3+, an enhanced food image segmentation model based on DeepLabv3+ with a ResNet50 backbone. Our approach involves adding a middle layer in the decoder path and SimAM after each extracted backbone feature layer. Secondly, we present CamerFood10, the first food image dataset specifically designed for sub-Saharan African food segmentation. It includes 10 classes of the most consumed food items in Cameroon. On our dataset, mid-DeepLabv3+ outperforms benchmark convolutional neural network models for semantic image segmentation, with an mIoU (mean Intersection over Union) of 65.20%, representing a +10.74% improvement over DeepLabv3+ with the same backbone.

Evaluation of Acceptability, Functionality, and Validity of a Passive Image-Based Dietary Intake Assessment Method in Adults and Children of Ghanaian and Kenyan Origin Living in London, UK.

Accurate estimation of dietary intake is challenging. However, whilst some progress has been made in high-income countries, low- and middle-income countries (LMICs) remain behind, contributing to critical nutritional data gaps. This study aimed to validate an objective, passive image-based dietary intake assessment method against weighed food records in London, UK, for onward deployment to LMICs. Wearable camera devices were used to capture food intake on eating occasions in 18 adults and 17 children of Ghanaian and Kenyan origin living in London. Participants were provided pre-weighed meals of Ghanaian and Kenyan cuisine and camera devices to automatically capture images of the eating occasions. Food images were assessed for portion size, energy, nutrient intake, and the relative validity of the method compared to the weighed food records. The Pearson and Intraclass correlation coefficients of estimates of intakes of food, energy, and 19 nutrients ranged from 0.60 to 0.95 and 0.67 to 0.90, respectively. Bland-Altman analysis showed good agreement between the image-based method and the weighed food record. Under-estimation of dietary intake by the image-based method ranged from 4 to 23%. Passive food image capture and analysis provides an objective assessment of dietary intake comparable to weighed food records.

Deep neural network for food image classification and nutrient identification: A systematic review.

Technology impacts human life in both the aspects such as positive and negative, which helps in better communication and eliminating geographical boundaries. However, social media and mobile devices may lead to severe health conditions such as sleep problems, depression, obesity, etc. A systematic review is conducted to analyze health issues by tracking food intake by considering positive aspects using Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA) Guidelines. The major scientific databases (such as Web of Science, Scopus, and IEEE explore) are explored to search the image recognition and analysis articles. The search query is applied to the databases using keywords like "Food Image," "Food Image Classification," "Nutrient Identification," "Nutrient Estimation," and using "Machine Learning," etc. 771 articles are extracted from these databases, and 56 are identified for final consideration after rigorous screening. A few investigations are extracted based on available food image datasets, hyperparameters tuning, a technique used, performance metrics, and challenges of Food Image Classification (FIC). This study discusses different investigations with their proposed FIC and nutrient estimation solution. Finally, this intensive research presents a case study using FIC and object detection techniques to estimate nutrition with food image analysis.

Large-scale transient peri-ictal perfusion magnetic resonance imaging abnormalities detected by quantitative image analysis.

Epileptic seizures require a rapid and safe diagnosis to minimize the time from onset to adequate treatment. Some epileptic seizures can be diagnosed clinically with the respective expertise. For more subtle seizures, imaging is mandatory to rule out treatable structural lesions and potentially life-threatening conditions. MRI perfusion abnormalities associated with epileptic seizures have been reported in CT and MRI studies. However, the interpretation of transient peri-ictal MRI abnormalities is routinely based on qualitative visual analysis and therefore reader dependent. In this retrospective study, we investigated the diagnostic yield of visual analysis of perfusion MRI during ictal and postictal states based on comparative expert ratings in 51 patients. We further propose an automated semi-quantitative method for perfusion analysis to determine perfusion abnormalities observed during ictal and postictal MRI using dynamic susceptibility contrast MRI, which we validated on a subcohort of 27 patients. The semi-quantitative method provides a parcellation of 3D T1-weighted images into 32 standardized cortical regions of interests and subcortical grey matter structures based on a recently proposed method, direct cortical thickness estimation using deep learning-based anatomy segmentation and cortex parcellation for brain anatomy segmentation. Standard perfusion maps from a Food and Drug Administration-approved image analysis tool (Olea Sphere 3.0) were co-registered and investigated for region-wise differences between ictal and postictal states. These results were compared against the visual analysis of two readers experienced in functional image analysis in epilepsy. In the ictal group, cortical hyperperfusion was present in 17/18 patients (94% sensitivity), whereas in the postictal cohort, cortical hypoperfusion was present only in 9/33 (27%) patients while 24/33 (73%) showed normal perfusion. The (semi-)quantitative dynamic susceptibility contrast MRI perfusion analysis indicated increased thalamic perfusion in the ictal cohort and hypoperfusion in the postictal cohort. Visual ratings between expert readers performed well on the patient level, but visual rating agreement was low for analysis of subregions of the brain. The asymmetry of the automated image analysis correlated significantly with the visual consensus ratings of both readers. We conclude that expert analysis of dynamic susceptibility contrast MRI effectively discriminates ictal versus postictal perfusion patterns. Automated perfusion evaluation revealed favourable interpretability and correlated well with the classification of the visual ratings. It may therefore be employed for high-throughput, large-scale perfusion analysis in extended cohorts, especially for research questions with limited expert rater capacity.

The effects of school gardens on fruit and vegetable consumption at school: A randomized controlled trial with low-income elementary schools in four U.S. states

This randomized controlled trial examines the effects of a school garden intervention on children’s fruit and vegetable (FV) consumption at school over two years. We randomly assigned schools to the intervention group that received gardens and related curriculum (n = 24) or to the waitlist control group that received gardens and curriculum at the conclusion of the study (n = 22). Children in second, fourth, and fifth grade at baseline (n = 2767) in low-income schools (n = 46) in four U.S. States (Arkansas, Iowa, New York, and Washington) participated. The intervention comprised gardens for each classroom; a curriculum focused on nutrition, plant science, and horticulture, including activities and FV tasting sessions; resources for the school that addressed topics such as soil contamination and food safety; an implementation guide focused on issues related to planning, planting, and maintaining the garden through the year, engaging volunteers, summer gardening, building community capacity, and sustaining the gardening program. FV consumption was measured by photographing lunches before and after children ate, for 2–3 days, at baseline and at each of 3 subsequent periods of data collection during the intervention. FV consumption was calculated using Digital Food Image Analysis. Among children in the intervention, fruit consumption and low-fat vegetable consumption increased from pre-garden baseline to post-garden more than among control group children. Garden intervention fidelity (GIF) also predicted changes in dietary intake, with more robust interventions showing a stronger effect than weaker interventions. GIF-lessons was a particularly potent predictor of change in dietary intake. School gardens modestly increase children’s FV consumption at school.

Transmittance Multispectral Imaging for Reheated Coconut Oil Differentiation

Oil reheating has a significant impact on global health due to its extensive consumption, especially in South Asia, and severe health risks. Nevertheless, food image analysis using multispectral imaging systems(MISs) has not been applied to oil reheating analysis despite their vast application in rapid food quality screening. To that end, the paper discusses the application of a low-cost MSI to estimate the 'reheat cycle count classes' (number of times an oil sample is recursively heated) and identify 'critical classes' at which substantial changes in the oil sample have materialized. Firstly, the reheat cycle count class is estimated with Bhattacharyya distance between the reheated and a pure oil sample as the input. The classification was performed using a support vector machine classifier that resulted in an accuracy of 83.34 % for reheat cycle count identification. Subsequently, an unsupervised clustering procedure was introduced using a modified spectral clustering (SC) algorithm to distinguish critical classes under reheating. In addition, laboratory experiments were performed to ascertain the ramifications of the reheating process with a chemical analysis. The chemical analysis of the coconut oil samples used in the experiment coincided with the chemical analysis results and was statistically significant (p < 0.05). Accordingly, the proposed work closes the gap for using multispectral imaging for oil reheating and proposes a novel algorithm for unsupervised detection of critical property changes in the oil. Hence, the proposed research work is significant in its practical implications, contribution to food image analysis, and unsupervised classification mechanisms.

A Comprehensive Survey of Image-Based Food Recognition and Volume Estimation Methods for Dietary Assessment.

Dietary studies showed that dietary problems such as obesity are associated with other chronic diseases, including hypertension, irregular blood sugar levels, and increased risk of heart attacks. The primary cause of these problems is poor lifestyle choices and unhealthy dietary habits, which are manageable using interactive mHealth apps. However, traditional dietary monitoring systems using manual food logging suffer from imprecision, underreporting, time consumption, and low adherence. Recent dietary monitoring systems tackle these challenges by automatic assessment of dietary intake through machine learning methods. This survey discusses the best-performing methodologies that have been developed so far for automatic food recognition and volume estimation. Firstly, the paper presented the rationale of visual-based methods for food recognition. Then, the core of the study is the presentation, discussion, and evaluation of these methods based on popular food image databases. In this context, this study discusses the mobile applications that are implementing these methods for automatic food logging. Our findings indicate that around 66.7% of surveyed studies use visual features from deep neural networks for food recognition. Similarly, all surveyed studies employed a variant of convolutional neural networks (CNN) for ingredient recognition due to recent research interest. Finally, this survey ends with a discussion of potential applications of food image analysis, existing research gaps, and open issues of this research area. Learning from unlabeled image datasets in an unsupervised manner, catastrophic forgetting during continual learning, and improving model transparency using explainable AI are potential areas of interest for future studies.

Explainable deep learning ensemble for food image analysis on edge devices

Food recognition systems recently garnered much research attention in the relevant field due to their ability to obtain objective measurements for dietary intake. This feature contributes to the management of various chronic conditions. Challenges such as inter and intraclass variations alongside the practical applications of smart glasses, wearable cameras, and mobile devices require resource-efficient food recognition models with high classification performance. Furthermore, explainable AI is also crucial in health-related domains as it characterizes model performance, enhancing its transparency and objectivity. Our proposed architecture attempts to address these challenges by drawing on the strengths of the transfer learning technique upon initializing MobiletNetV3 with weights from a pre-trained model of ImageNet. The MobileNetV3 achieves superior performance using the squeeze and excitation strategy, providing unequal weight to different input channels and contrasting equal weights in other variants. Despite being fast and efficient, there is a high possibility for it to be stuck in the local optima like other deep neural networks, reducing the desired classification performance of the model. Thus, we overcome this issue by applying the snapshot ensemble approach as it enables the M model in a single training process without any increase in the required training time. As a result, each snapshot in the ensemble visits different local minima before converging to the final solution which enhances recognition performance. On overcoming the challenge of explainability, we argue that explanations cannot be monolithic, since each stakeholder perceive the results’, explanations based on different objectives and aims. Thus, we proposed a user-centered explainable artificial intelligence (AI) framework to increase the trust of the involved parties by inferencing and rationalizing the results according to needs and user profile. Our framework is comprehensive in terms of a dietary assessment app as it detects Food/Non-Food, food categories, and ingredients. Experimental results on the standard food benchmarks and newly contributed Malaysian food dataset for ingredient detection demonstrated superior performance on an integrated set of measures over other methodologies.

Progressive Kernel Extreme Learning Machine for Food Image Analysis via Optimal Features from Quality Resilient CNN

Recently, food recognition has received more research attention for mHealth applications that use automated visual-based methods to assess dietary intake. The goal is to improve the food diaries by addressing the challenges faced by existing methodologies. In addition to the classical challenge of the absence of rigid food structure and intra-class variations, food diaries employing deep networks trained with pristine images are susceptible to quality variations in real-world conditions of image acquisition and transmission. Similarly, existing progressive classifiers that use visual features via a convolutional neural network (CNN) classify food categories and cannot detect food ingredients. We aim to provide a system that selects the optimal subset of features from quality resilient CNNs and subsequently incorporates the parallel type of classification to tackle such challenges. The first progressive classifier recognizes food categories, and its multilabel extension detects food ingredients. Following this idea, after extracting features from the quality resilient category and ingredient CNN models by fine-tuning it on synthetic images generated using the novel online data augmentation method random iterative mixup. Our feature selection strategy uses the Shapley additive explanation (SHAP) values from the gradient explainer to select the best features. Then, novel progressive kernel extreme learning machine (PKELM) is exploited to cater to domain variations due to quality distortions, intra-class variations, and so forth, by remodeling the network structure based on activity value with the nodes. PKELM extension for multilabel classification detects ingredients by employing a bipolar step function to process test output and then selecting the column labels of the resulting matrix with a value of one. Moreover, during online learning, the PKELM novelty detection mechanism can label unlabeled instances and detect noisy samples. Experimental results showed superior performance on an integrated set of measures for seven publicly available food datasets.

Improving Food Detection For Images From a Wearable Egocentric Camera

Diet is an important aspect of our health. Good dietary habits can contribute to the prevention of many diseases and improve overall quality of life. To better understand the relationship between diet and health, image-based dietary assessment systems have been developed to collect dietary information. We introduce the Automatic Ingestion Monitor (AIM), a device that can be attached to one’s eye glasses. It provides an automated hands-free approach to capture eating scene images. While AIM has several advantages, images captured by the AIM are sometimes blurry. Blurry images can significantly degrade the performance of food image analysis such as food detection. In this paper, we propose an approach to pre-process images collected by the AIM imaging sensor by rejecting extremely blurry images to improve the performance of food detection.

DeepFood: Food Image Analysis and Dietary Assessment via Deep Model

Food is essential for human life and has been the concern of many healthcare conventions. Nowadays new dietary assessment and nutrition analysis tools enable more opportunities to help people understand their daily eating habits, exploring nutrition patterns and maintain a healthy diet. In this paper, we develop a deep model based food recognition and dietary assessment system to study and analyze food items from daily meal images (e.g., captured by smartphone). Specifically, we propose a three-step algorithm to recognize multi-item (food) images by detecting candidate regions and using deep convolutional neural network (CNN) for object classification. The system first generates multiple region of proposals on input images by applying the Region Proposal Network (RPN) derived from Faster R-CNN model. It then indentifies each region of proposals by mapping them into feature maps, and classifies them into different food categories, as well as locating them in the original images. Finally, the system will analyze the nutritional ingredients based on the recognition results and generate a dietary assessment report by calculating the amount of calories, fat, carbohydrate and protein. In the evaluation, we conduct extensive experiments using two popular food image datasets - UEC-FOOD100 and UEC-FOOD256. We also generate a new type of dataset about food items based on FOOD101 with bounding. The model is evaluated through different evaluation metrics. The experimental results show that our system is able to recognize the food items accurately and generate the dietary assessment report efficiently, which will benefit the users with a clear insight of healthy dietary and guide their daily recipe to improve body health and wellness.

Yum-Me: A Personalized Nutrient-Based Meal Recommender System.

Nutrient-based meal recommendations have the potential to help individuals prevent or manage conditions such as diabetes and obesity. However, learning people’s food preferences and making recommendations that simultaneously appeal to their palate and satisfy nutritional expectations are challenging. Existing approaches either only learn high-level preferences or require a prolonged learning period. We propose Yum-me, a personalized nutrient-based meal recommender system designed to meet individuals’ nutritional expectations, dietary restrictions, and fine-grained food preferences. Yum-me enables a simple and accurate food preference profiling procedure via a visual quiz-based user interface and projects the learned profile into the domain of nutritionally appropriate food options to find ones that will appeal to the user. We present the design and implementation of Yum-me and further describe and evaluate two innovative contributions. The first contriution is an open source state-of-the-art food image analysis model, named FoodDist. We demonstrate FoodDist’s superior performance through careful benchmarking and discuss its applicability across a wide array of dietary applications. The second contribution is a novel online learning framework that learns food preference from itemwise and pairwise image comparisons. We evaluate the framework in a field study of 227 anonymous users and demonstrate that it outperforms other baselines by a significant margin. We further conducted an end-to-end validation of the feasibility and effectiveness of Yum-me through a 60-person user study, in which Yum-me improves the recommendation acceptance rate by 42.63%.

Digital Food Image Analysis as a Measure of Children’s Fruit and Vegetable Consumption in the Elementary School Cafeteria: A Description and Critique

ABSTRACTMeasuring the dietary intake of large samples of children is a challenge. This article describes a protocol for capturing a large number of digital images of lunch trays in a school cafeteria environment, addresses some of the advantages and challenges of this procedure, and examines the validity of Digital Food Image Analysis (DFIA; beta version) as a tool to quantify the amount of fruits and vegetables (FV) that elementary schoolchildren are served and that they consume. The beta version of DFIA is a valid method for assessing fruit and vegetable consumption and, with modest correction, the amount of FV served.

FOOD IMAGE ANALYSIS: THE BIG DATA PROBLEM YOU CAN EAT!

Six of the ten leading causes of death in the United States can be directly linked to diet. Measuring accurate dietary intake, the process of determining what someone eats is considered to be an open research problem in the nutrition and health fields. We are developing image-based tools in order to automatically obtain accurate estimates of what foods a user consumes. We have developed a novel food record application using the embedded camera in a mobile device. This paper describes the current status of food image analysis and overviews problems that still need to be addressed.

Segmentation of Food Images using Morphological Operations with Watershed Segmentation Technique

Person across the world are now becoming more conscious about their health and weight. Accuracy of the calorie measuring system is depending on food image analysis. The proper analysis of an image is based on image segmentation technique and is one of the important steps in image analysis. Multiple image segmentation techniques exist for extracting requires objects from an image. Different image segmentation techniques based on edge detection, region growing, based on clustering and thresholding are applied on food images but these techniques are not useful for segmenting the overlapped regions of food. Watershed segmentation technique is used to solve the problem of over segmentation and segmenting small regions which are overlapped to each other but this segmentation results are not more accurate for increasing accuracy of segmentation result. In this paper work hybrid model is proposed by combining Morphological Operations and Watershed Segmentation technique (MOWS) to segment overlapped regions of food with their shape. Watershed Segmentation and MOWS are implemented in MATLAB and Analysis is done based on their PSNR values. Analysis shows that proposed hybrid model MOWS is given more accurate result than watershed segmentation.

A Mobile Food Record For Integrated Dietary Assessment.

This paper presents an integrated dietary assessment system based on food image analysis that uses mobile devices or smartphones. We describe two components of our integrated system: a mobile application and an image-based food nutrient database that is connected to the mobile application. An easy-to-use mobile application user interface is described that was designed based on user preferences as well as the requirements of the image analysis methods. The user interface is validated by user feedback collected from several studies. Food nutrient and image databases are also described which facilitates image-based dietary assessment and enable dietitians and other healthcare professionals to monitor patients dietary intake in real-time. The system has been tested and validated in several user studies involving more than 500 users who took more than 60,000 food images under controlled and community-dwelling conditions.

EFFICIENT SUPERPIXEL BASED SEGMENTATION FOR FOOD IMAGE ANALYSIS.

In this paper, we propose a segmentation method based on normalized cut and superpixels. The method relies on color and texture cues for fast computation and efficient use of memory. The method is used for food image segmentation as part of a mobile food record system we have developed for dietary assessment and management. The accurate estimate of nutrients relies on correctly labelled food items and sufficiently well-segmented regions. Our method achieves competitive results using the Berkeley Segmentation Dataset and outperforms some of the most popular techniques in a food image dataset.

Study and Analysis of Image Segmentation Techniques for Food Images

As a person across the world is now becoming more caring about their health and weight. Many systems are available to measure the calorie intake from the food images before and after eating. Accuracy of the calorie measuring system is depending on food image analysis. The proper analysis of an image is based on image segmentation technique and is one of the important steps in image analysis. Multiple image segmentation techniques exist for extracting requires objects from an image. In this paper different image segmentation techniques based on Edge Detection, Morphological Operation, Threshold and Clustering based techniques studied and analyzed for segmenting different food images. Different image segmentation techniques implemented in MATLAB and then analyzed.

ANALYSIS OF FOOD IMAGES: FEATURES AND CLASSIFICATION.

In this paper we investigate features and their combinations for food image analysis and a classification approach based on k-nearest neighbors and vocabulary trees. The system is evaluated on a food image dataset consisting of 1453 images of eating occasions in 42 food categories which were acquired by 45 participants in natural eating conditions. The same image dataset is used to test the classification system proposed in the previously reported work [1]. Experimental results indicate that using our combination of features and vocabulary trees for classification improves the food classification performance about 22% for the Top 1 classification accuracy and 10% for the Top 4 classification accuracy.