Optimizing a Transformer-based network for a deep learning seismic processing workflow

Abstract

StorSeismic is a recently introduced model based on the Transformer network to adapt to various seismic processing tasks through its pretraining and fine-tuning strategy. In the original implementation, StorSeismic utilized a sinusoidal positional encoding and a conventional self-attention mechanism, both borrowed from natural language processing (NLP) applications. For seismic processing they provided good results, but also hinted at limitations in efficiency and expressiveness. We propose modifications to these two key components, by utilizing relative positional encoding and low-rank attention matrices as replacements for the standard ones. The proposed changes are tested on processing tasks applied to a realistic Marmousi and offshore field data as a sequential strategy, starting from denoising, direct arrival removal, multiple attenuation, and finally root-mean-squared velocity ( VRMS) prediction for normal moveout (NMO) correction. We observe faster pretraining and competitive results on the fine-tuning tasks and, additionally, fewer parameters to train compared to the standard model.