The accuracy of O/X mode separation in vertical ionograms directly determines the quality of pattern discrimination results and metrics, which is of great significance to ionospheric research. It’s extremely complicated to separate the O/X mode of the vertical ionograms because of the environmental noise, interference and the time-varying dispersion characteristics of the ionosphere itself. In this paper, we propose a method for separating the O/X mode signal in vertical ionograms based on an improved U-shaped Encoder–decoder network, named VIS-UNet. Our model is based on the Encoder-decoder architecture. It introduces the residual convolution to avoid network performance degradation and utilizes the attention module to improve the attention of the signal characteristic. In addition, we design an adaptive loss function to expedite the convergent speed of the model. Experimental results show that our model performs better than the baselines for the task of the O/X mode signal separation: (1) The method in this paper has low requirements on the vertical ionospheric sounding system, and the ionograms obtained by the single-channel vertical ionospheric sounding system can realize the separation of O/X mode signal at the pixel level. (2) It has strong universality. It is insensitive to the signal integrity and the ionospheric pattern of the vertical ionograms.(3) It performs better for the separation task. The MIOU of the O/X mode separation task reaches 91.97% and the performance is significantly improved compared with the existing methods.