Histo-fetch – on-the-fly processing of gigapixel whole slide images simplifies and speeds neural network training

Abstract

BackgroundTraining convolutional neural networks using pathology whole slide images (WSIs) is traditionally prefaced by the extraction of a training dataset of image patches. While effective, for large datasets of WSIs, this dataset preparation is inefficient. MethodsWe created a custom pipeline (histo-fetch) to efficiently extract random patches and labels from pathology WSIs for input to a neural network on-the-fly. We prefetch these patches as needed during network training, avoiding the need for WSI preparation such as chopping/tiling. Results & conclusionsWe demonstrate the utility of this pipeline to perform artificial stain transfer and image generation using the popular networks CycleGAN and ProGAN, respectively. For a large WSI dataset, histo-fetch is 98.6% faster to start training and used 7535x less disk space.