Multispectral panoptic segmentation: Exploring the beach setting with worldview-3 imagery

Abstract

Panoptic segmentation is a recent and powerful task that tackles individual object recognition (“things”) and multiple backgrounds (“stuff”) simultaneously. Remote sensing studies with panoptic segmentation are still restricted and recent, with great application perspectives. In this sense, we propose the first multispectral panoptic segmentation study, considering the “thing” and “stuff” classes in the beach scenario and evaluating different sets of spectral bands. Our methodology included developing a dataset with 3800 (3200 for training, 300 for validation, and 300 for testing) with 128 × 128 spatial dimensions and eight spectral bands considering fourteen classes (6 “thing” and 8 “stuff” classes). We used WorldView-3 images from Praia do Futuro, Fortaleza, and pan-sharpening to improve spatial resolution. Five different spectral band configurations were considered: (1) all eight bands, (2) R G B + N I R 1 + N I R 2 , (3) R G B + N I R 1 , (4) R G B + N I R 2 , and (5) only RGB. The model training used the Panoptic-FPN architecture with the same hyperparameter settings considering three backbones (ResNeXt-101, ResNet-101, and ResNet-50). The best result considered the ResNeXt-101 with all spectral bands. However, the results from the first four configurations were very similar, and the RGB alone was the only configuration with significantly lower results. We also evaluated 15 semantic segmentation models for a benchmark comparison for the Beach Dataset. We show in visual results that even though the semantic models may be precise, they fail at identifying unique targets, especially in crowded locations such as the beach. The panoptic segmentation allowed a necessary detailing and counting of tourist infrastructures and mapping of other background features, establishing an essential tool for inspecting beach areas. • First panoptic segmentation study in the beach setting. • First panoptic segmentation study comparing spectral band compositions. • First panoptic segmentation study using multispectral WorldView-3 imagery. • Novel beach dataset suitable for instance, semantic and panoptic segmentation.