ABSTRACT Spectral X-ray computed tomography (SXCT) is able to provide richer spectral information than conventional X-ray CT, thereby improving the capability of discriminating materials in segmentation tasks. However, segmentation algorithms for conventional CT images often lose their performance in material segmentation with SXCT images due to the spectrum-dependent attenuation properties of X-rays. Recently, deep neural networks (DNNs) have demonstrated their effectiveness in segmentation tasks. However, processing SXCT with DNNs requires significant memory resources and may take significant computation time. To address these issues, this paper introduces a simulation study towards realising deep-learning-based SXCT image segmentation. The proposed method employs a convolutional neural network (CNN) that selectively utilises the significant spectrum channels for accurate material segmentation. The proposed method builds upon a prior study on adaptive CT reconstruction, but to further enhance the performance, we introduce a self-attention-based spectrum-selection module. During training, the selection module assigns importance weights to spectrum channels based on the attention scores. Once the network has been trained, the proposed method can obtain satisfactory segmentation results from projection images at one or a combination of several spectrum channels recommended by the selection module. The experimental results demonstrate that the proposed method achieves a significant reduction in computation time while maintaining the segmentation quality.