Up to now, most existing steganalytic methods were designed for grayscale images, and are not suitable for the color images that are widely used in social networks. In this paper, we design a universal color image steganalysis network (called UCNet) for the spatial and JPEG domains. The proposed method includes preprocessing, convolutional, and classification modules. To preserve the steganalytic features in each color channel, the preprocessing module first separates the input image into three channels based on the corresponding embedding spaces (i.e., RGB in the spatial domain, and YCbCr in the JPEG domain), and then extracts the image residuals with 62 fixed high-pass filters. Finally, all truncated residuals are concatenated for subsequent analysis, rather than adding them together in the first layer as in existing CNN-based steganalyzers. To accelerate network convergence and effectively reduce the number of parameters, the convolutional module contains three carefully designed types of layers with different shortcut connections and group convolution structures, to further learn the high-level steganalytic features. In the classification module, we employ global average pooling and a fully connected layer for classification. We conduct extensive experiments on ALASKA II to demonstrate that the proposed method can achieve state-of-the-art results that are comparable with other modern CNN-based steganalyzers (e.g., SRNet and LC-Net) in both the spatial and JPEG domains, with relatively few memory requirements and short training times. Furthermore, we also provide some necessary descriptions and carry out numerous ablation experiments to verify the rationality of the network design.