From the abundance perspective: Multi-modal scene fusion-based hyperspectral image synthesis

Abstract

Nowadays, data is of paramount importance for artificial intelligence. However, collecting real-world hyperspectral images (HSIs) with desired characteristics and diversity can be prohibitively expensive and time-consuming, leading to the data scarcity issue in HSI, and further limiting the potential of deep learning-based HSI applications. Existing work to tackle this issue fails to generate abundant, diverse, and reliable synthetic HSIs. This work proposes a multi-modal scene fusion-based method that diffusion from the abundance perspective for HSI synthesis, termed MSF-Diff. Concretely, highlights involve: (1) Synthesis in low-dimensional abundance space, other than original high-dimensional HSI space, greatly releases the difficulty; (2) Integration of multi-modal data greatly enriches the diversity of spatial distribution that the model can perceive; (3) Incorporation of the unmixing concept ensures that the generated synthetic HSI has reliable spectral profiles. The proposed research can generate a vast amount of HSI with a rich diversity in various categories and scenes, closely resembling realistic data. It plays a pivotal role in ensuring that the model produces reliable results and can be trusted for real-world applications. The code is publicly available at https://github.com/EtPan/MSF-Diff.