An Intelligent Vision-Based Nutritional Assessment Method for Handheld Food Items

Abstract

Dietary assessment has proven to be an important tool to evaluate the dietary intake of patients with diabetes and obesity. The traditional approach of accessing the dietary intake is to conduct a 24-hour dietary recall (24HR), a structured interview designed to obtain information on food categories and volume consumed by the participants. Due to the unconscious biases in this kind of self-reporting approach, many research studies have explored the use of vision-based approaches to provide accurate and objective assessments. Despite the promising results of food recognition by deep neural networks, there still exist several hurdles in deep learning-based food volume estimation ranging from domain shift between synthetic and raw 3D models, shape completion ambiguity and, most importantly, lack of large-scale paired training dataset. Therefore, this paper proposed an intelligent nutritional assessment approach via weakly-supervised point cloud completion. It aims to close the reality gap in 3D point cloud completion tasks and address the targeted challenges by proposing the Point Cloud Auxiliary Classifier GAN (PC-ACGAN), a novel and robust point cloud completion architecture that can be trained in a weakly-supervised manner. Then the volume can be easily estimated from the completed representation of the food. In addition to advantages with regard to efficiency, another major merit of our system is that it can be used to estimate the volume of handheld food items without requiring the constraints including placing the food items on a table or next to fiducial markers, which facilitate the implementation on both wearable and handheld cameras. Comprehensive experiments have been carried out on major benchmark datasets and self-constructed volume-annotated dataset respectively, in which the proposed PC-ACGAN demonstrates comparable results with several strong fully-supervised baseline methods and shows superior completion ability in handling food volume estimation.