HSIFoodIngr-64: A Dataset for Hyperspectral Food-Related Studies and a Benchmark Method on Food Ingredient Retrieval

Abstract

Listen

Translate article

Food-related issues have attracted increasing attention recently due to its various applications in our daily life, for example, restaurant service and dietary assessment, which are essential for human health management and understanding food characteristics. Many existing datasets are proposed for various food-relevant tasks, e.g., food detection and segmentation. However, most of the datasets are labelled with only dish-level annotations and lack detailed information for the corresponding ingredients. Hyperspectral imaging (HSI), which can explore the emissive characteristics of different objects with a long region of spectral bands, is of great potential for food and ingredient analysis. In the present work, a new food image dataset HSIFoodIngr-64 containing 3,389 pairs of HSI and RGB images with 21 dish classes and 64 ingredient categories was established. It is the first HSI-based food image dataset, of which all images are labelled with dish-level and ingredient-level annotations and corresponding pixel-wise ingredient masks. Therefore, our dataset can be applied to different food-centric tasks. Furthermore, this research proposed a benchmark method on the food ingredient retrieval task, which consists of two sub-networks called IPSN for ingredient prediction sub-network and ERSN for edge refinement sub-network. Extensive experiments were conducted on HSIFoodIngr-64 dataset and the effectiveness of our proposed method was verified. HSIFoodIngr-64 dataset is expected to provide a new perspective on food analysis and inspire more efforts on various food-related issues. We have made the HSIFoodIngr-64 dataset public at: https://doi.org/10.7910/DVN/E7WDNQ.